#JShell project for Tj-Bot!
1 messages ยท Page 6 of 1
not before tests
alr we can do some mvc tests to make sure endpoints work as expected
some unit tests to keep some logic safe, like the id validation
not the time to integration test
The problem is that any more rework/improvment may break stuff, so we need tests to be sure everything is alright
well that's the problem
id validation is like the least important test
we need to test if the sessions work
because that's the thing that can break
We need them
Because otherwise there is no way to know if we didn't break something
That's the next priority after the bug fix
Yea that's a problem
as long as we don't have a way to test the endpoints automatically
doing a refactor is too dangerous
Note that spring allows to do unit with doing requests to endpoints
so it might be easier than we think
u mean without* ?
if we test endpoints, we'll need to mock deps
check the WebMockTest class
the last one
nooooo
don't be a meanie :(
it's not constructive
instead, suggest improvements 
and if they make sense after discussing with us, go implement 'em!
im gonna share something in few mins
stay tunned
i still dnt know why id comes from client
it could lead to duplicates
nevermind
what do u think of this ?
beautiful
the project is called JShellPlaygroundBackend
somebody may ask backend for what ?
here's finished version
yes ?
it doesn't matter
the bot is out of the scope of this project
You could improve both text and simplify them
Also it doesn't really fetch anything, it just send the code to execute
but u said it could retrieve snippets ?
it could
the diagram has to show all major aspects
but there is no notion to retrieve a session, it doesn't make sense
ah
yes
and it could run 2 or 3
what are 2 or 3 ?
containers
it will select the correct one by name
yes?
so i make sense to show 2 containers on diag
the code is complicated and i think it has serious issues
yes
alr
its not though
if you find parts of the code complicated, ask for clarification
me and ala both understand it and can follow it very well
cuz u ve been on it since long time
but that wont help newcomers
so u think everything is fine
Saying the code is complicated is an overstatement
I have seen much more complex codebase
This is nothing
i didn't say that
me too
thts the fact
what do you find complicated about the jshellsessionservice @bright lark ?
JShellService
what part about it is complicated?
public JShellService(DockerService dockerService,
JShellSessionService sessionService,
String id,
long timeout,
boolean renewable,
long evalTimeout,
long evalTimeoutValidationLeeway,
int sysOutCharLimit,
int maxMemory,
double cpus,
String cpuSetCpus,
String startupScript)
so these are parameters that the docker client needs
and some of them are for us
so if we passed a "DockerConfig" instead, is the class now no longer complicated?
the same problem will happen, it'll be a complex record, builder, factory, whatever you decide
but it's all loaded for us
I already have a local branch ready for this btw
I am just waiting for the tests before I try to create a pr
you don't need to use the constructor yourself
it's just loaded from that config file
i need to be able to read it
the actual complicated part of the constructor
is not the parameters
try {
String containerId = dockerService.spawnContainer(maxMemory, (long) Math.ceil(cpus),
cpuSetCpus, containerName(), Duration.ofSeconds(evalTimeout), sysOutCharLimit);
PipedInputStream containerInput = new PipedInputStream();
this.writer = new BufferedWriter(
new OutputStreamWriter(new PipedOutputStream(containerInput)));
InputStream containerOutput =
dockerService.startAndAttachToContainer(containerId, containerInput);
reader = new BufferedReader(new InputStreamReader(containerOutput));
writer.write(sanitize(startupScript));
writer.newLine();
writer.flush();
checkContainerOK();
startupScriptSize = Integer.parseInt(reader.readLine());
} catch (Exception e) {
LOGGER.warn("Unexpected error during creation.", e);
markAsDead();
throw new DockerException("Creation of the session failed.", e);
}
it's this
i didn't say tht
For my defense, I wasn't the one who wrote this ๐
that's okay, i already blame istan
it should be moved in a service class
JShellService
it's already in the service class
nah, it's a model class
why does it need an id?
well
not really
controller -> JShellSessionService -> JShellService -> DockerService
Here we are in the 3rd one
the left has a dependency to the right
in this order
the problem is that JShellSessionService and JShellService are coupled
if we can remove that and handle it in the controller it'll be a little neater
we don't need a model
the only model we need is for Session's
and that can branch out of it needed
sessions are JShellService
thts what the code says
but JShellService is used to run the command commands??
it's got methods like eval
readResult
I already have a fix for that too, but it will wait for later, not a problem currently
private synchronized JShellService createSession(SessionInfo sessionInfo) {}
createSession returns JShellService
there is no class with tht name
ik
actually i feel super confused between these 2
JShellService and JShellSessionService
JShellSessionService handles sessions
JShellService is one session
if we used game driven development, it would be pretty cool actually, the JShellSessionService could be like an "action handler" and we can use a "game loop" to handle cleanups, changing states etc.
and players == JShellSession's
the code becomes easier to understand as humans
trust me, it's not readable (these 2 core ~~service ~~classes)
Ah and also
i trust you, that you find it not readable
for the reason why JShellService isn't called "session"
it's because JShellService has no knowledge of being a session, it's JShellSessionService which make it a session
they have logic, they can't be model
yeah so insense it's indirectly like: java class JShellSession { JShellSession(JShellService service) {} }
even though that's not the literal code
you can model the ideas in your head and visually in your mind see the pattern being used
even if not explict in the code
i know, but they could be refactored
maybe you read code differently, but if you can associate the design with somebody familar to you, you'll understand it
so they become : 1 service class + 1 model class
๐คทโโ๏ธ
but not yet
lmao
and also it doesn't make it easier to read
you can't begin to refactor something you don't understand
like what do you want to refactor ? the bazillion ? parameters in JShellService ?
It won't change much in term of readability
which is where stuff like UML would help
if we had all the classes represented on a diagram
we can visually pen and paper refactor
You can create a class diagram in intellij btw
and all come to an agreement
yeah but it sucks ass
i don't think that's useful for the problems we've had
the tests you've got so far are more around eval
and docker tests are not easy
@bright lark I would like that you do them, is it possible ?
You do not need to understand the code, just think of it as a black box testing, just know how to use the api
yes but that's the problem
yeah maybe firas could do it
The thing that has the more chances to fail is docker
it would deffo help him a lot
as well help with gather confidence in jshell
but honestly, starting again from scratch would be the best approach instead of pick and choose refactoring (just imo)
we can start re-implementing the code just in a more managed way
once we fully break down the system
I don't find a problem with the system
which is fine if the community can maintain it
we should make it a more TJ project rather than an Ala project
but it's unapproachable because of the lack of understanding people have, it's more harder because it's a new project
unlike TJ Bot which is at least based on common tooling (JDA)
but the mess we're in now is because of improper planning which was unavoidable anyway
u mean integration test ?
yes @bright lark those are the most beneficial atm
i dnt see this feasible
why?
what complexity do you see?
i already have a good solution in mind
we have all the methods needed to help with assertions
thoses injections
@sterile crest https://spring.io/guides/gs/testing-web
^^
I was just about to mention
we use Spring and it has all the tooling available
it's not about tooling
then?
it's about a not necessary complexity
what complexity though
how does this answer the question?
(i didn't fully read docker service yet)
what question
what complexity..
Don't read it, your eyes will bleed
@bright lark you just need to do black box testing, this is unrelated to anything you said
never herd of this type
it should be straight forward : ```
@BeforeAll
void startSpring() {}
@Test
void someTest() {
// either post to the API
// call the JShellService directly
// something else
assert(...);
}
It's when you don't assume the implementation, you only look at the input/output
tht means mocking
no it doesn't
That's not how spring works but yes
spring is flexible, but i assume in integration tests, i need to load a spring context with required beans, like the app does in reality
@SpringBootTest
class SmokeTest {
@Autowired
private HomeController controller;
@Test
void contextLoads() throws Exception {
assertThat(controller).isNotNull();
}
}
so this example let's you do your spring magic
assuming the server is running:
@SpringBootTest(webEnvironment = WebEnvironment.RANDOM_PORT)
class HttpRequestTest {
@LocalServerPort
private int port;
@Autowired
private TestRestTemplate restTemplate;
@Test
void greetingShouldReturnDefaultMessage() throws Exception {
assertThat(this.restTemplate.getForObject("http://localhost:" + port + "/",
String.class)).contains("Hello, World");
}
}
i know how to test in spring
this is a better example
it's not going to be like this
after context is loaded for tests, we need to pull img
then perform api call to run a container
it happens under the hood
let me check but i'm sure the code pulls the image if not present
public String spawnContainer(long maxMemoryMegs, long cpus, @Nullable String cpuSetCpus,
String name, Duration evalTimeout, long sysoutLimit) throws InterruptedException {
String imageName = "togetherjava.org:5001/togetherjava/jshellwrapper";
boolean presentLocally = client.listImagesCmd()
.withFilter("reference", List.of(imageName))
.exec()
.stream()
.flatMap(it -> Arrays.stream(it.getRepoTags()))
.anyMatch(it -> it.endsWith(":master"));
if (!presentLocally) {
client.pullImageCmd(imageName)
.withTag("master")
.exec(new PullImageResultCallback())
.awaitCompletion(5, TimeUnit.MINUTES);
}
hold on
i see image name and port written raw in code
wtf
yeah i know
oh shit
ye it's maybe the next one
Please think about soleving the test problem instead of stopping every 3 minutes to detail that you don't need to stop to
i'm trying to answer all your concerns and questions now @bright lark so that you're not hesistent to approach the problem
i'm able to do it
but i dnt like it this way
tht's all
and knowing tht my motivation is my only purpose to play with this project
there's 2 types of integration testing you can do
starting from Spring or starting directly with DockerService
which way ?
DockerService is the easy one to test
1st, we simplify the core service classes
Spring, it's the most important one
We can't simplify without tests

yeah we need tests otherwise we won't know if we broke anything during refactoring
we already learned this when i refactored the project
but we wrote tests shortly after learning
so while refactoring is great, we're all for it - we might shoot ourselves in the foot
unless we have tests to protect us from these accidental mistakes
we need a lot in place like tests, diagrams etc. so we know what we're doing
it's a collaborative process, nothing should be on one person
it's just nice that you stepped up to helps us, we appreciate it a lot
ok then
but we also have to make sure that we're doing the right things for the project
otherwise, i would have replaced the entire solution with an AWS Lambda ages ago 
alright, let imagine now tests are set
and pass without errors
what would be next ? !
take things step by step
sure we can plan the next 10 steps but that's not good use of our time right now
why no body else did the tests ?!
because istan was alone and when ala jumped on he was alone
Note that I did tests
for the wrapper that is
because tests for the wrapper were important at the time
but now
what is important
is the tests for the api
I usually put test depending of how much something can break
wrapper that I made was very fragile
so I added tests
the api wasn't fragile back then
because it was much more simple
but it is no longer the case (since istan added the docker service)
so we need tests
@bright lark
@sterile crest
whole project is made by ala, with some help with docker stuff 
istannen wrote the previous jshell bot
yea docker is istan, the rest is me
yea it's very small
it's just interacting with loads of moving parts
but the files themselves are not massive either
DockerService is 157 LoC, JShellService 293 LoC
And JShellSessionService?
JShellSessionService 133 LoC
the entire API repo is practically less than 1000 LoC
it's not about lines
yes it isn't otherwise we wouldn't need tests
but it shows that the project is small and you can easily understand a few lines of code
JShellSessionService
private void initScheduler() {
scheduler = Executors.newSingleThreadScheduledExecutor();
scheduler.scheduleAtFixedRate(() -> {
List<String> toDie = jshellSessions.keySet()
.stream()
.filter(id -> jshellSessions.get(id).shouldDie())
.toList();
LOGGER.info("Scheduler heartbeat, sessions ready to die: {}", toDie);
for (String id : toDie) {
JShellService service = jshellSessions.get(id);
if(service.isMarkedAsDead()) {
try {
jshellSessions.remove(id).close();
} catch (Exception ex) {
LOGGER.error("Unexpected exception for session {}", id, ex);
}
} else {
service.markAsDead();
}
}
}, config.schedulerSessionKillScanRateSeconds(),
config.schedulerSessionKillScanRateSeconds(), TimeUnit.SECONDS);
}
you telling me you really can't understand what's going on here?
private synchronized JShellService createSession(SessionInfo sessionInfo)
throws DockerException {
// Just in case race condition happens just before createSession
if (hasSession(sessionInfo.id())) {
return jshellSessions.get(sessionInfo.id());
}
if (jshellSessions.size() >= config.maxAliveSessions()) {
throw new ResponseStatusException(HttpStatus.TOO_MANY_REQUESTS,
"Too many sessions, try again later :(.");
}
LOGGER.info("Creating session : {}.", sessionInfo);
JShellService service = new JShellService(dockerService, this, sessionInfo.id(),
sessionInfo.sessionTimeout(), sessionInfo.renewable(), sessionInfo.evalTimeout(),
sessionInfo.evalTimeoutValidationLeeway(), sessionInfo.sysOutCharLimit(),
config.dockerMaxRamMegaBytes(), config.dockerCPUsUsage(), config.dockerCPUSetCPUs(),
startupScriptsService.get(sessionInfo.startupScriptId()));
jshellSessions.put(sessionInfo.id(), service);
return service;
}
this is the only other main logic in that class
sure the jumping between classes makes it a bit harder
but look at the code at a bird eye view
and it's all of a sudden not a scary project ๐
and it's why it's a simple codebase to refactor
we just to organise the methods a little
The thing is that there are like only three things that happen in this method
- case for race condition
- failfast
- new JShellService
Like that's all, this is how simple it is
the naming ala uses can be a bit confusing tho ๐
when he says buzz words like "race conditon"
but insense
it's the same as
String add(String s) {
if(list.contains(s)) {
return list.get(s);
}
return list.add(s);
}```
It actually is
The only reason I have this if is in case two threads arrive here at the same time
but yea, it's just a if
yeah but it's just a guard though if we disassociated the "why"
yea
you'd always normally check if it exists before creating
there are 2 guard clause
then logic is just one new + one put to map
how to build the img from test class
try some experimentations
part of the fun is problem solving, it's why we're developers ๐
aim to enjoy yourself
and don't stress without getting involved, outside of us 3, there's the entire server who can help us out if we got stuck
nothing needs to be done in 1 day either so take your time, jshell was rushed to delivery but we can take our time because all things considered, it's not going to harm us - we want to aim for quality after all and be proud of the feature
i did
i wonder why we don't have a repo for images ?
it's for free on Docker Hub, and as project is public, we can have a public repo for images
docker images are meant to be stored on a server repo, like we do for code on github
this can significantly simplifies integreation tests
which is my current task
if we put the wrapper image from a repository, with testconainers we can pull image and run it easily
you can pull it from our self hosted docker repo
the source doesn't matter, it's like doing mavenLocal if this was a gradle dep
i know but in tests, that requires building the img beforeall
and deleting it afterall
by launching gradle commands
and docker commands
yeah but you can do it in gradle itself
test {
dependsOn docker
}
or whatever the name is
so when you run gradle test or whatever command, it'll sort the image out
locally running in intellij, you'll have image pulled already
so gradle is able to run a task
ofc?
when i run test task ?
yeah
duh? it's one of the primary usecases
what do you think happens when you do gradle build?
it compiles, runs tests, runs spotless etc.
it's been while i didn't use gradle
there's also the doFirst doLast blocks you can do too
or if you want to ensure a better ordering mustRunAfter, shouldRunAfter
etc
loads of config available anyway, check out the docs
that should help
LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext loaded from file:/C:/.../.gradle/caches/8.7/generated-gradle-jars/gradle-api-8.7.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml: org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext
java.lang.IllegalArgumentException: LoggerFactory is not a Logback LoggerContext but Logback is on the classpath. Either remove Logback or the competing implementation (class org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext loaded from file:/C:/.../.gradle/caches/8.7/generated-gradle-jars/gradle-api-8.7.jar). If you are using WebLogic you will need to add 'org.slf4j' to prefer-application-packages in WEB-INF/weblogic.xml:
org.gradle.internal.logging.slf4j.OutputEventListenerBackedLoggerContext
when i run the test
this is my class
@SpringBootTest
@AutoConfigureMockMvc
public class JShellApiTests {
private static final Logger LOGGER = LoggerFactory.getLogger(Main.class);
@Autowired
private MockMvc mockMvc;
@Test
public void test() {
String codeSnippet = "2 + 2";
try {
mockMvc.perform(MockMvcRequestBuilders.post("/jshell/eval/test")
.contentType(MediaType.APPLICATION_JSON)
.content(codeSnippet))
.andExpect(MockMvcResultMatchers.status().isOk())
.andExpect(MockMvcResultMatchers.jsonPath("$.status").value(SnippetStatus.VALID));
} catch (Exception e) {
LOGGER.error("something went wrong: {}", e.getMessage());
}
}
}
Good stuff, I'm sure you can understand how to fix the error ๐
ive been searching for 1 hour
tried to exclude like this :
test {
dependsOn ':JShellWrapper:jibDockerBuild'
exclude("spring-boot-starter-logging")
exclude("logback-classic")
}
nothing changed
the problem is due to the logger in Main class ig
ive got to learn more about Gradle..
Any advancement ?
still learning gradle tricks
yesterday hadn't time for jshell
however i still have weird error
^
wby?
@sterile crest can you help him ?
also did you change the buil.gradle?
yes ive added a control
to check if img exist, otherwise build it
i have also to add a check when tests end, remove the img
i dunno about jib
if it could remove the img
along the way i learn about gradle custom tasks
show it please
task buildDockerImage {
group = 'Docker'
description = 'builds jshellwrapper as docker image'
dependsOn jibDockerBuild
}
task removeDockerImage {
group = 'Docker'
description = 'removes jshellwrapper image'
commandLine 'docker', 'rmi', '-f', imageName
}
test {
dependsOn buildDockerImage
finalizedBy removeDockerImage
}
this is not the final config
intellij wants me to use tasks.register()
in addition to some other warnings, it's confusing me
here is a final version for tasks
tasks.register('buildDockerImage') {
group = 'Docker'
description = 'builds jshellwrapper as docker image'
dependsOn jibDockerBuild
doFirst{
println('creating jshellwrapper image...')
}
doLast{
println('docker image is ready for use')
}
}
tasks.register('removeDockerImage', Exec) {
group = 'Docker'
description = 'removing jshellwrapper image'
commandLine 'docker', 'rmi', '-f', imageName
doLast{
println('docker image has been removed')
}
}
tasks.named('test') {
dependsOn tasks.named('buildDockerImage')
doFirst {
try {
println 'Running JShellAPI tests...'
} catch (Exception e) {
println 'JShellAPI tests failed'
tasks.named('removeDockerImage').get().execute()
throw e
}
}
doLast {
println 'JShellAPI tests completed.'
}
finalizedBy tasks.named('removeDockerImage')
}
looks nice in execution tbh
time to commit this
it's a simple error in nature, spring ships with its own logger and there's another version conflicting on the path
it'll be from one of the other dependencies
so what you need to do is analyse the dependency tree and exclude it
the question is
why this doesn't happen when running the app ?
it could also be because gradle's internal logging is conflicting with another lib
because you're not using mockito, junit or any of the testing support during running? ?
they're all libraries being added to the classpath during test
i dnt think so
ive tried logging in others apps like this one
never saw this error
lul
maybe im wrong
u are
push your code to a branch, i'll look at it later
im free in an hour or so
the main idea is to exclude the conflicting path during the test step
alright
i found it
testImplementation ('org.springframework.boot:spring-boot-starter-test') {
configurations {
all {
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
exclude group: 'ch.qos.logback', module: 'logback-classic'
exclude group: 'org.apache.logging.log4j', module: 'log4j-to-slf4j'
}
}
}
why i can't open a branch on github ?
np im forking the repo
alright, ive pushed changes on a forked repo
what's done in JShellApi subproject :
- custom gradle tasks for tests โ
- creating a first class for tests โ
feel free to give me feedback
// ----
what's next :
- set some integration tests for /jshell/eval
- maybe will add some unit tests to protect some methods from breaking
okay but you need to link me to your fork
why did you create a new task called test @bright lark ?
anyways
works for me
once I removed ur garbage
tasks.named('test') {
dependsOn tasks.named('buildDockerImage')
doFirst {
try {
println 'Running JShellAPI tests...'
} catch (Exception e) {
println 'JShellAPI tests failed'
tasks.named('removeDockerImage').get().execute()
throw e
}
}
doLast {
println 'JShellAPI tests completed.'
}
finalizedBy tasks.named('removeDockerImage')
}
delete all of this
and replace it with
test.dependsOn buildDockerImage
testImplementation('org.springframework.boot:spring-boot-starter-test') {
configurations {
all {
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
exclude group: 'ch.qos.logback', module: 'logback-classic'
exclude group: 'org.apache.logging.log4j', module: 'log4j-to-slf4j'
}
}
}
also all of this, replace with
testImplementation('org.springframework.boot:spring-boot-starter-test') {
configurations {
all {
exclude group: 'ch.qos.logback', module: 'logback-classic'
}
}
}
you can remove testImplementation gradleTestKit() not needed
i need to remove the img when test is done
why
okay
also, it's useful to remove the img if test fails
in that case
I'm certain this isn't needed either
tasks.register('buildDockerImage') {
group = 'Docker'
description = 'builds jshellwrapper as docker image'
dependsOn jibDockerBuild
doFirst{
println('creating docker image...')
}
doLast{
println('docker image is ready for use')
}
}
test.dependsOn jibDockerBuild
import java.time.Instant
plugins {
id 'org.springframework.boot' version '3.2.5'
id 'io.spring.dependency-management' version '1.1.5'
id 'com.google.cloud.tools.jib' version '3.4.2'
id 'com.github.johnrengelman.shadow' version '8.1.1'
}
dependencies {
implementation project(':JShellWrapper')
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'com.github.docker-java:docker-java-transport-httpclient5:3.3.6'
implementation 'com.github.docker-java:docker-java-core:3.3.6'
testImplementation('org.springframework.boot:spring-boot-starter-test') {
exclude group: 'ch.qos.logback', module: 'logback-classic'
}
annotationProcessor "org.springframework.boot:spring-boot-configuration-processor"
}
def imageName = 'togetherjava.org:5001/togetherjava/jshellbackend:master' ?: 'latest';
jib {
from.image = 'eclipse-temurin:21'
to {
image = imageName
auth {
username = System.getenv('ORG_REGISTRY_USER') ?: ''
password = System.getenv('ORG_REGISTRY_PASSWORD') ?: ''
}
}
container {
mainClass = 'org.togetherjava.jshellapi.Main'
setCreationTime(Instant.now().toString())
}
}
shadowJar {
archiveBaseName.set('JShellPlaygroundBackend')
archiveClassifier.set('')
archiveVersion.set('')
}
test.dependsOn jibDockerBuild
test.doLast {
project.exec {
commandLine 'docker', 'rmi', '-f', imageName
}
}
there
updated your build.gradle
removed all the extra crap you added for no reason

what
my impl is better in terms of readability
as it provides grouping, descriptions and logs
those are useful, when performing a build on any env
i tried to put myself in a position of a beginner
i can clean but not like as u did
it's too short
also, it seems that tasks.register() is recommended
recommended by who
by gradle docs
how
your impl has a million lines of boilerpalte
see you can't read the docs
it's telling you how to create a new task
but the tasks already exist
i can clean
your feedback is great
so we can set something at the middle
sure you can get your PR merged by somebody else then
maybe @timber mirage will do it
what you're doing is:
the eqv of
void print(String s) {
System.out.println("running print");
println(s);
System.out.println("finished print");
}```
it makes no sense
That's not what Wazei is saying. Changes should be made for a good reason, and add value. At first glance this PR adds verbosity, with at most an initial beginner ramup easement?
i can get behind creating a gradle task to delete an image but not the other bits
It's not that the sentiment isn't appreciated. But ideally before changing existing functionality, start it from a discussion/ticket point of view, and don't start from a code point of view. A lot of decisions were made for a specific reason.
if we a set a task for removing, it would be obvious to set a task to create
and grouping those, is helpful
as for the println() i can reduce or get rid of em
Is a delete task useful? It's easy to manage yourself.
nope not useful, the only reason i can see for it is that we have a create task so lets have a delete but that isn't normally the task for a build tool anyway (in our context)
That last bit was indeed my initial thought.
if it's not useful, then how should it be ?!
There are a lot of ways to manage the removal of images, but that's usually not managed by your build file.
ideally, if the test needs setup, you'd do it in the @BeforeAll step
i was going to do tht
but it was u who suggested to go with gradle build file
so i followed that suggestion and found it feasible
also, as the build command is in the build file, it was good to keep those controls in same place
it's not best practise due to inconvincence for maintainability, if we decide gradle isn't for us, switching to maven would be a bigger chore
so java testing code will be less verbose and focused on the api
so you'd want to keep a lot of heavy work outside of gradle
but you know, test.dependsOn jibDockerBuild this is a convincence as we already have an existing task
these are gradle specific traps to be careful about and one of the reasons people prefer maven
it's super easy to start doing a lot of logic in your gradle build scripts
?
just some general knowledge sharing, but you know, people in other places to some crazy things ๐
as for tomorrow, we need to push the image on a respository
and use testcontainers. and get rid of those temporary solutions
it's published to our local repo afaik
local
it's accessible by the VPS only I think? ? togetherjava.org:5001
i dnt get this
I might be wrong though, maybe ala or marko can clarify
ok
tried this
and it works, but it doesn't remove the img when test fails
and it doesn't remove it when it passes !
id have to create img each time i run tests ?

how this looks fine to u ?
hold on
i think i made a mistake
yeah
it deletes the img when test passes, but doesn't remove it when it fails :/
test {
dependsOn jibDockerBuild
doLast {
project.exec {
commandLine 'docker', 'rmi', '-f', imageName
}
}
}
waiting for ur answer @sterile crest
^^
that keyword requires an object
that means i need to create a task ?
yeah
task removeDockerImage {
doLast {
project.exec {
commandLine 'docker', 'rmi', '-f', imageName
}
}
}
test.finalizedBy removeDockerImage
like that
and you get your removeDockerImage task
ahaa
we get back to custom task
without a creation task, it'd be ambiguous to contributors to understand this
ok
๐คทโโ๏ธ
im gonna group them under "integration test"
or maybe simply "docker"
here we go
tasks.register('buildDockerImage') {
group = 'docker'
description = 'builds jshellwrapper as docker image'
dependsOn jibDockerBuild
}
tasks.register('removeDockerImage', Exec) {
group = 'docker'
description = 'removes jshellwrapper image'
commandLine 'docker', 'rmi', '-f', imageName
}
test {
dependsOn tasks.named('buildDockerImage')
finalizedBy tasks.named('removeDockerImage')
}
You don't need the tasks.named to ref it
def taskBuildDockerImage = tasks.register('buildDockerImage') {
group = 'docker'
description = 'builds jshellwrapper as docker image'
dependsOn jibDockerBuild
}
def taskRemoveDockerImage = tasks.register('removeDockerImage', Exec) {
group = 'docker'
description = 'removes jshellwrapper image'
commandLine 'docker', 'rmi', '-f', imageName
}
test {
dependsOn taskBuildDockerImage
finalizedBy taskRemoveDockerImage
}
Yeee

Kinda iffy about the first one because it's a task to rename and existing one but fineee
Btw
Use the short form like I showed in my task
Instead of tasks.registrr just task name { }
why not ?
Well it doesn't matter because all the docs are there for the registered one
read this
Why
it is recommended to register tasks with register() api
I don't see the recommendation
But do what you want
I don't wanna spend time going back and fourth on this
You're unblocked anyway
i have a weird error
2024-08-10T16:02:23.809+01:00 INFO 21152 --- [pool-2-thread-1] o.t.j.service.JShellSessionService : Scheduler heartbeat: sessions ready to die: []
2024-08-10T16:02:23.810+01:00 WARN 21152 --- [o-auto-1-exec-1] o.t.jshellapi.service.JShellService : Unexpected error during creation.
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Head \"https://togetherjava.org:5001/v2/togetherjava/jshellwrapper/manifests/master\": no basic auth credentials"}
i need credentials to spawn the container ?
i dunno why this didn't happen when i run the app
i think the spawnContainer() in dockerservice is wrong
ah i get it
the image name is different
i made a mistake in gradle build file, the command should beJShellWrapper:jibDockerBuild
yes
jib -> create the image in the server, needs cradentials
jibDockerBuild -> create the image in local
@bright lark
btw why do you need to delete the image ?
locally
when test is done
if i keep it. When i rerun test, new img is created
im gonna have to inject jshelwrapper image name from root gradle file
alright, ive finished on test so far
it took time honestly but it works
and i think it would be a good idea to review my changes in a first PR
so next tests are rebased on it
๐ im gonna send pr
do you at least one working test ?
so it's pointless to delete it ?
yes
wdym
also can you chose different kind of config in the tests ?
cuz tht's the scenario
?
- build img
- start api server
- perform api call
- run container from img
- gets code result
- shutdown container
- delete image
but why do you need to delete the image ?
and ?
also i told u if keep it
everytime i run tests, new image is added
this is cumbersome
why ?
i only need 1 img
possible
but i see that deletion is better
why ?
it's close to real scenario
alright, consider performin the tests on ci/CD
it would be better to delete it once tests are done
wdym
so about the tests
- testing eval results
- testing single eval
- testing eval edge case behavior (while(true), crash, etc)
- testing the sessions (sending several code one after each other and seeing it works)
- testing all kinds of timeouts
- testing snippets
- testing shutdown (especially edge cases)
and probably more
i know
but id submit my pr now
to review the approach
on which we can rebase for other tests
we can keep the issue open and have multiple prs
yes
@compact holly @sterile crest can you review this please ? โ Since you know way better than me how gradle works
@bright lark what is this https://github.com/Together-Java/JShellPlaygroundBackend/blob/3502cb76912531531a5259b047e1eb5cb6f8089c/JShellAPI/src/main/java/org/togetherjava/jshellapi/rest/ApiEndpoints.java ?
constants class
for the endpoints
That you don't use ?
i used in tests
Then why is it here ?
hmm
it should be under /test
ye mb
well no
i forgot to synchronize it with the paths in controller
the idea is to keep paths same between controllers and test classes
@bright lark how does additional-spring-configuration-metadata.json work?
it helps to know more about the custom properties
how?
ah i see
it shows an error
it's so the IDE can enforce a type
what a trash way to do, spring sucks ๐ฆ
it's trash because it's just used for IDE linting, it doesn't need to be included in the final built jar
Request failed: Too Many Requests. Too many sessions, try again later :(.
JShell has been down for awhile now
What's up?
Just gonna leave it in a broken state or what?
hello
@sterile crest as for diagrams, i ve first reviewed alathreon's PR in which he added an overview diagram
i can add some diagrams, i already did some
but i dunno if that would be acceptable
make more
i did, i just kept them on my laptop
but as i recall, marko said that doing such , will make the project look like a real product dedicated for commercial
and that's not the objective
while my intention is to provide clear overview to us and newcomers
also, there is something else before the diagrams, writing some documentation about the usecases and actors
following Alistar cockburn model
.
@sterile crest I am talking about this pr https://github.com/Together-Java/JShellPlaygroundBackend/pull/49
It's been a month that it is waiting for a review
@sterile crest I merged it and created and merge a pr and master
And @compact holly just to tell you that the jshell backend just got updated
I also would like that @sterile crest and @compact holly review this PR, since @bright lark is editing a lot of config, and you are much better than me when it comes to gradle
Looks like it's still having issues
The warning logs are going crazy with the NPE that's being fired
what are you talking about ?
org.togetherjava.tjbot.features.help.MarkHelpThreadCloseInDBRoutine
WARN
unable to mark thread as close with id :1269585872718790667
java.lang.NullPointerException: Cannot invoke "net.dv8tion.jda.api.entities.channel.concrete.ThreadChannel.getIdLong()" because "threadChannel" is null
at org.togetherjava.tjbot.features.help.HelpThreadLifecycleListener.handleArchiveStatus(HelpThreadLifecycleListener.java:86)
at org.togetherjava.tjbot.features.help.MarkHelpThreadCloseInDBRoutine.lambda$updateTicketStatus$1(MarkHelpThreadCloseInDBRoutine.java:66)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.togetherjava.tjbot.features.help.MarkHelpThreadCloseInDBRoutine.updateTicketStatus(MarkHelpThreadCloseInDBRoutine.java:63)
at org.togetherjava.tjbot.features.help.MarkHelpThreadCloseInDBRoutine.runRoutine(MarkHelpThreadCloseInDBRoutine.java:49)
at org.togetherjava.tjbot.features.system.BotCore.lambda$scheduleRoutines$7(BotCore.java:199)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/jav...
this ?
this has nothing to do with jshell
ah yeah, I was looking at it then stopped looking at the class it was erroring in
log blindness ๐ฆ
Whatsp
๐คท
no issues with jshell so far?
@compact holly -_-
i mean bugs, is it stable?
No issue so far, but please review firas PR
you review it, it's your project
if you don't understand the tech then why did you choose it 
It's not mine, it's ours
I didn't chose gradle
If it was maven, yes I could review it
but i don't know gradle
learn it!
and I am not the one how chose it
the time firas's PR has been around, you could have mastered gradle 
I need someone who is experienced in it, it doesn't matter if I learn it, we need someone who is experienced in gradle
nah, it's fine, it's not on ala to do everything
it's our project, and everyone should contribute what they can
now, it's true that it's only us here, and nobody knows or really cares about this project
but that's the story for another time
we do have to think about maintainability and long-term sustainability
the thing is, you can use your initative. Does everything still work? Do the changes look sensible? Check there isn't a "rm -r System32" in there and if it looks good, give it a tick
I didn't even review your PRs i just ticked them for example 
it's also fine to wait for domain experts to review, or codeowners
if you are not comfortable reviewing the code
it's actually preferred, since we care about quality more than just pushing changes (that are not rm system32 or keyloggers)
...seriously please
I am expecting that any pr are at elast correctly reviewed
and we care about stability more than the new features
so don't lie in your reviews
LGTM
it would be tragic if adding tests (attempt to make software more stable), actualy takes down our production service
then i'm not the best person to ask in the current time
i just came from home from the airport
@sterile crest @compact holly if nobody review it or tell that they will review it before saturday, i'll merge it
that's fair at the current project state, but not the best approach overall
what I really dislike is jshell being broken for a month, that shouldn't ever happen on our production services
at if it happens, that feature should be disabled, with a friendly explanation
so at least there should be pipeline to make sure everything builds, and few smoke screen/integration tests that make sure that basic functionality is there (i'm aware pr is about that)
and you want actual reviews not only to ensure stability and maintainability (which is crucial for the project)
but also to engege community with the project, and share knowledge
if project is a black box made by a single member, not documented enough, doesn't have good guides and easy to do issues.. nobody will join
because it's a daunting task, that would require a week for an experienced dev to delve deep into it and reproduce and fix a bug
Yes but that also takes a lot of time
yup, time and effort
like everything in life.. 
but anyway, whatever we do, it should at least be stable enough
broken features should be disabled until fixed
or just revert them to the previous working state
It currently works great
yup, everything else is seconodary
so just merging random commits you are unsure about, that are not tested or at least built locally
is not the best approach for keeping it working great
Yea but here I have no choice, will I will test it locally, but that's not what is needed
yeah, but it's the most important step
if you are sure it works, sure merge it, fuck it
rest of the codebase is not reviewed either
I wouldnt say so, 95% is not
and wazei admitted to fake reviewing, and sabotaging project from the inside 
yeah, it's creative 
oh hello Oh boy
What a news!
What going on
so if nobody else review your pr, I'll merge it this friday
se we will be able to advance
huh U can't see it?
?
If u can see it, then u can review it
Ah! Then ask me in pr to explain it somewhere
Probably in readme
It's good tht we got this reaction now in order to do what should be done
As for what marko said, id agree to write some docs ideally with some diagrams
Ala did some progress in this and i reviewed his pr some weeks ago
Also, i did some docs to usecases with other sort of diagrams but kept it locally as i didn't want to share before we merge ala's pr
can you focus on tests instead please ?
I think i did
And we agreed tht my pr about testing is a first to refine and agree on a strategy for the rest
I dunno, why would i write N tests if perhaps they'll be rejected?
wdym
I did one test rather than many to see if the approach i followed is accepted
If it's not ok or lacks something
Then same reviews apply to next tests
Whether it's me or others
well, after reading your pr, the problem seems to be that the tests don't scale well, you are using constants everywhere for the variables to send and the tests has a lot of code just to test a 2+2
you should refactor it so you only need minimal lines of code
also, I would want a second test where you a second request to see if your way of doing the tests can handle state
i think those 2 problems are the biggest problems that i see in your pr
Good notes! Please write in pr
and Ill handle godwilling
You're right about the constants some should be inside the test method. As for the endpoint constants i see them useful to maintain sync between controllers and test classes
I admit tht i was a bit hasty before my latest commit
But i can fix in pr review process (which is made for tht)
I did
No
I made it clear when I did the review, the PRs were stab at the dark fixes to see if the problem goes away
At least the main big problem has been resolved which is containers are being destroyed
Any other PR that pops up, idc about, I have no need to review. They are not for me and I'm only interested in bug fixes
@compact holly here you go first good first issue https://github.com/Together-Java/JShellPlaygroundBackend/issues/53
very cute 
@timber mirage I swear I see "session running" but for the life of me can't find who invoked the command 
I just wanted to see who was using it and where and what
Consider creating an issue to show these metrics?
Just look at the issue just above your message ๐
I didn't understand that issue
what is off ?
nope
Can I do it for jshell
What's that ?
what do you actually need
I don't want to introduce huge systems that someone has to understand, maintain.. and all that
It's small effort
infra stuff should be as simple as possible
depends what you do with it, but it's pretty big boy tool
How will it work ? Is it a TJ feature or JShell backend feature ?
when you want to integrate all these tools on a fancy graph, centralized logging and monitoring
you add prometheus
Think of a class called metric that contains a single Map<String, String> and the rest is abstracted away not ever needing to be touched
No
That's adding complexity
I just want metrics
On jshell
ye, if you can add a monitoring and metrics system on the vps
Note that there is already a pr about this
There isn't
Feel free to improve it
That's some noob version
It can be minimal, i.e. just a Map<String, String>
Then a single method to pop it into graphana
Or whatever we want
Currently, with this pr, it returns some info
but there are two improvments to make :
- also show docker stats about any container starting with session_
yeah, but graphs are sexy
- also show deleted sessions
data is just.. some numbers
but that's all
I have 2 things on my TODO list for TJ
AI based top helper check
And graphana for jshell
But once graphana is done, it's a plug and play for every other service
i have no idea how many people are using it
or any metrics about it
well, to be honest, I have no metrics at all on this server
wait, do you want to add graphana on tj bot project, or jshell backend project ?
Idk who uses it and where, I'm under the impression there's fake logging

on tjbot*
Everything tbh
Even the website
but it's true that we need stats on tj bot
I was asking this since a long time ago
I am pretty sure I made several #server-suggestions about it, and I am not the only one
And I am pretty sure people already did PR for it
it would be really nice to actually know what the fuck is happening, how are users using our software
and so on
but we all agree on that
It's a solved problem though, we have all the knowledge to do it
so wazei
Just about waiting till somebody can be bothered
You're implementing it wrong
with the pr I linked earlier, I will improve it a bit
Your pr isn't compatible
so you can just call it to feed graphana
why ?
Problem we have is that it's 2 repos and we don't have a common library
So c&p but whatever
we can have a common library
Yeah I'm scared of future problems with that approach
My PR is about monitoring jshell state
What you want is to collect stats of anything so they can be checke later
Right @sterile crest ?
me as well, has to be maintained as such


?