data-science-and-ml

[torpid linden]

thanks mate

[dreamy elbow]

Does anyone here have experience making a .py file work through streamer.bot? I made a coach character/personality, and want him to be interactable with my chat. Everything LOOKS right, but it's not working

[hollow cobalt]

If anyone has experience in LLM classification combined with instruction fine-tuning instruction-response pair creation it would really help if we could speak about some formats using metadata-driven data insertion it would be helpful with a project I'm doing atm?

[dry panther]

bro this server dead even nobody i can talk with

[hollow cobalt]

Do you still need help?

[dreamy elbow]

Yeah but I’m away from the computer now. I do really want to finish that project.

[hollow cobalt]

IDM helping

[dreamy elbow]

Cool! I’ll be home from work tomorrow 6pm est. I’ll let you know!

[hollow cobalt]

sweet

[hollow fossil]

what is that

[hollow fossil]

oh ic

im learning about neural networks right now and Im lost
like in a neural network you have so many components you can tweak, so many layers you can add so many parameters you can tweak
my models perform at like 60% whereas others perform close to 90%
a lot of tutorials ive seen just tell you how to run a neural network, its basic components etc etc
what they dont teach is how to structure a NN layer by layer, how you can decide which layer should include what so on and so forth
any advice regarding this ?

https://www.geeksforgeeks.org/deep-learning/image-classification-using-cifar-10-and-cifar-100-dataset-in-tensorflow/

[jaunty helm]

what they dont teach is how to structure a NN layer by layer, how you can decide which layer should include what so on and so forth
it's because you don't, beyond stuff like "convolution layer makes sense for image data"
how deep should it go, how wide should it go, etc. a lot of the time comes down to try it and see if it's better

[hollow fossil]

i see, i kind of get your point but as I said due to the sheer amount of stuff you can tweak here its inefficent to try so many diff combinations

i want a more direct way to get to it, rather than jsut groping in the dark

[jaunty helm]

copy the architecture of someone who has worked on a similar task
for example, you're doing image classification - then you can start by copying resnet50 and go from there

on the other hand, if you can afford it, it's still just hyperparameters that you can automatically select through libraries like optuna or something

a little more systematic way is just start with a large parameter count that overfits, then start fighting it through e.g. reducing parameters or dropout until you hit the sweet spot (or the reverse, start small and grow until sweet spot)
obviously this wouldn't tell you other decisions like the type of layer, activation, regularization, etc. though

[orchid light]

Output from my llm (in the pretraining stage on only 5.5B tokens for now)

[Generated 128 tokens in 0.85s (150.2 tok/s)]```

[waxen kindle]

nice, how long did it take to train it and on what hardware ?

[gritty vessel]

Anyone used optuna here?

How it works with large datasets?

or with model like taking around 40mins for an epoch

So like even if we give 5 epoch its gonna take 200mins for a single trial

[waxen kindle]

it will

[gritty vessel]

Than for like 50trials it will be so much time

[waxen kindle]

if you don't use a gpu, you will be able to parallelize tasks

[gritty vessel]

I have gpu and its using around 10gb

but heres the thing what I saw like i trained one set of parameters on Unet with conv3d single channel and depth of 8

[waxen kindle]

idk how it works if you need a gpu, I feel like multiple tasks (that are threads) will use it at the same time and that will break everything

not sure tho

[gritty vessel]

around 12000 sequences took 40mins for each epoch

[gritty vessel]

but training is slower on cpu right?

[waxen kindle]

yes

I think you need 1 GPU for each task

[gritty vessel]

damn

Unet with conv3d is taking like 10x time when comapred to unet with conv2d

let me read little about this

[gritty vessel]

optuna?

Well yeah but for huge dataset I am not able to understand how will it work

what you talking about then?

aah okay

I did not see that

looks interesting

which one you are using?

I mean which model you are using?

I see

Its a huge model

It should work fine with low reasoning ?

like reasoning is kind of that deep research thing right?

aah okay got it

follow ups if I am not wrong?

yes

well good thing happened

=== Trial 0 ===
Hyperparameters: loss=fcl, optimizer=RMSprop, lr=0.000038, scheduler=cosine
Epoch 1/2 | Train Loss: 0.932754 | Val Loss: 0.914847

all past iterations has min loss 7.something

but the thing is this is with 20% data only

I have seen this feature but don't ask me where 😵‍💫

[gritty vessel]

ignore this its wrong Ithink

[gritty vessel]

I have seen follow up questions before

like if I dont respond it like sends followups

[lapis sequoia]

Genuine question, and I'm brand new to this but what's it like being a datascientist? Does anyone use R code programming? Or is there a specific code most stick to as a data scientist?

[serene scaffold]

There's pretty much no consistency between companies, or even within companies, as to what a data scientist is or does

It would be unusual to use both python and R. The only people I know who use both learned R because python became ascendant, and they just still prefer to do certain things in R.

[calm cipher]

this is just a vague notion but I get the impression R is more common in non-CS fields, I've known several people studying psychology who are really familiar with it but never used Python before

I guess that makes sense if it's true, Python is nice and we have Pandas, but R is just data manipulation without the baggage of a whole programming environment to learn too

I would say if you have the option and you're interested in programming/CS, definitely learn Python first since it's more generally useful

and then picking up R if you need it shouldn't be too hard

[serene scaffold]

Yes, this is how it is.

[lapis sequoia]

😭 thank you all so much

[calm cipher]

If you go the Python route, learn Pandas for loading and manipulating data

also the statsmodels package, it's for advanced statistical modeling and was deliberately made to resemble statistical testing in R https://www.statsmodels.org/stable/index.html

I think they actually test its results against R's results to make sure the models work the same way

Also Scipy for simpler statistical tests

[sharp solstice]

guys should i learn transformative models over making my own lstm and gru/rnn in pytorch, i havn't worked with machinlearning/deeplearing in a while. ❤️

[tacit basin]

It depends. But transformer architecture like gpt is currently trending
https://karpathy.ai/zero-to-hero.html

[abstract wasp]

for those who have gotten job interviews for mle, how did did the interview process look like? did u guys have to complete leetcode style qs?
i hate leetcode but i hear some interviewers ask them 😭 thats the main thing thats stopping me rn from having much confidence 😭

[finite laurel]

[hollow fossil]

data_dir = get_file('flower_photos', dataset_url, cache_dir='.', untar=True)
data_dir = pathlib.Path(data_dir)
flower_names = ['roses','daisy','dandelion','sunflowers','tulips']
flower_images_dict = {}
for x in flower_names:
    print(x)
    flower_images_dict[x] = list(data_dir.glob('{}/*.jpg'.format(x)))
flower_images_dict```

[hollow fossil]

why is this code not working ?
for context im downloading an image set with 5 sub folders under it each with a set of images
but for some reason the glob is not detecting it at all and returns an empty list

[pallid vale]

a *.jpg glob will not go into subfolders. try a **/*.jpg one

example:

>>> list(p.glob("*.rst"))
[PosixPath('CONTRIBUTING.rst'), PosixPath('README.rst')]
>>> list(p.glob("**/*.rst"))
[PosixPath('CONTRIBUTING.rst'), PosixPath('README.rst'), PosixPath('docs/changelog.rst'), ...]

[half plank]

Hey , anyone know what logic or snippet or whatever is used to extract key statements from textfiles and pdf etcs , ( im making an llm which extracts keypoints from a type of documents and summarises the)

[past meteor]

Doing it with gpt 4.1 nowadays

Still in beta with us, but previously we had flow X and flow Y, users had to select what one they wanted which ... sucks because it's an implementation detail

We're trialing the selection between flow X and flow Y to be tool calls

[half plank]

Regx does extract some import stuff and is pre trained to an extent of format , but its not that accurate , it usually fails when facing 200-400 pages of data, the accuracy dips from 80% to 25/55%

[abstract wasp]

ive been doing all difficulties but i hope i get the ez ones ;-;

ty for the info

[calm cipher]

I'm confused about the premise, if you're using regex, that suggests the documents are in some structured format that contains important points in each document

But if that's true why do you need a LLM?

Or on the flip side, if the documents are unstructured such that you think machine learning is a good solution, what patterns are the regex looking for?

[past meteor]

It’s pretty cheap, what are you comparing it to?

How good is this?

[jaunty helm]

my memory might be poor, but I think I remember seeing groq's oss performing worse than other providers or smthn

[past meteor]

Howis the latency?

fwiw at work I stick to OpenAI

[green fiber]

how can i integrate the location thing which we see in zomato, swiggy apps?

like firstly i want the user to grant the location permission and then by using the user's location i have to show the nearby restaurants?

[half plank]

Maybe , idk , its been 16 hours since im doing , finally reaches a point where i can get the json-d output but now its not able to convert it to pdf , using file2pdf2

Btw switched to jupiter notebook

But now i have to make a freakin ur for accepting pdf , currently im using folder based input where u drop the pdf to n the folder

[orchid light]

Anyone knows how to finetune small pretrained models to behave like chat bot?

[serene scaffold]

what kind of model?

[orchid light]

Its an 116milion paramiter LLM

that i pre trained

But i`m having problems finetuning it

i tried blending different datasets

[serene scaffold]

start training it on text that's of the form of a dialog between a user and an agent.

[orchid light]

and its not working

[orchid light]

did it

  alpaca_cleaned: 20.0%
  dolly: 5.0%``` this is my blend

[serene scaffold]

remember to never say "it's not working" when asking for help with something code related--you always have to say exactly what you did, what that action was intended to do, and what actually happened.

[orchid light]

Ok im using LoRA with r set to 16 and im training on blend i sent above

[serene scaffold]

!unmute 733023093626241086

[arctic wedge]

:incoming_envelope: :ok_hand: pardoned infraction timeout for @orchid light.

[serene scaffold]

!paste

[arctic wedge]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[orchid light]

(nano_llm_env) kociolek@kociolek-OPTIMUS:~/Desktop/LLM$ python gen1.py   --lora_ckpt New_as1/lora_006000.pt   --lora_r 16   --lora_alpha 16   --lora_targets attn   --temperature 0.3   --top_p 0.8   --max_new_tokens 20
Using EOS id: 2 (tokenizer eos_id=2)
LoRA applied to 72 Linear layers | Trainable params: 1,106,688 / 117,517,200 (0.94%)
Loaded ln_f weight from LoRA checkpoint.

============================================================
:robot: Chat is ready (116M + LoRA)
Commands: /reset, /system <prompt>, quit/exit
============================================================

:bust_in_silhouette: You: Name an animal:
:robot: Bot: An animal is any of the animals that can be classified as mammals or birds.

:bust_in_silhouette: You: What color is grass?
:robot: Bot: Grass is a large genus of flowering plants and shrubs that have been used for food, shelter, and

:bust_in_silhouette: You: Say meow:
:robot: Bot: A grass is anything with a greenish-red color.

:bust_in_silhouette: You: Hi!
:robot: Bot: I'm glad you found this information useful.

:bust_in_silhouette: You: Hi!
:robot: Bot: Thank you for your question.

:bust_in_silhouette: You: HI!
:robot: Bot: If you have any other questions about something, feel free to ask.

:bust_in_silhouette: You: Hi!
:robot: Bot: You're welcome!

:bust_in_silhouette: You: Hello!
:robot: Bot: Is there anything else I can help you with?

:bust_in_silhouette: You: Hi!
:robot: Bot: No problem.

:bust_in_silhouette: You: !hI
:robot: Bot: Hi there!

:bust_in_silhouette: You: what are you?
:robot: Bot: Thanks for asking.

:bust_in_silhouette: You: so?
:robot: Bot: So far I've answered all of your questions.

:bust_in_silhouette: You: No?
:robot: Bot: Any more questions?

:bust_in_silhouette: You: Yes
:robot: Bot: Tell us if we need to get more information from us or if we want to recommend another site.

:bust_in_silhouette: You: What are you?
:robot: Bot: Are you happy to answer our questions.```

I think i pretrained a good model

So the problem is finetuning

[serene scaffold]

at least it's generating vaguely relevant text. this might be the best performance you can get with your constraints.
GPT-2 (which is before ChatGPT) had over a billion parameters.

[serene scaffold]

like I think this is how GPT-2 behaves if you try to use it as a chatbot

>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='gpt2')
>>> set_seed(42)
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)

[{'generated_text': "Hello, I'm a language model, a language for thinking, a language for expressing thoughts."},
 {'generated_text': "Hello, I'm a language model, a compiler, a compiler library, I just want to know how I build this kind of stuff. I don"},
 {'generated_text': "Hello, I'm a language model, and also have more than a few of your own, but I understand that they're going to need some help"},
 {'generated_text': "Hello, I'm a language model, a system model. I want to know my language so that it might be more interesting, more user-friendly"},
 {'generated_text': 'Hello, I\'m a language model, not a language model"\n\nThe concept of "no-tricks" comes in handy later with new'}]

[orchid light]

😔

[serene scaffold]

why sad?
you might have created the best possible model within your constraints

[orchid light]

Yeah but i just spend 4 days of training compute on it (on my rtx 4060 ti)

[serene scaffold]

GPT-3 trained on several V100s for 34 days

[orchid light]

Yeah but gpt 3 trained on 300 bilion tokens

and i trained on 10 bilion

[serene scaffold]

so you should have expected worse performance than GPT3
and it looks like you're currently on par with GPT2

[orchid light]

Yeah but its not a good resoult...

I trained on much better data

with modern architecture

and modern optimizers

[orchid light]

@serene scaffold ```👤 You: Hello!
🤖 Bot: Hey!

👤 You: How are you?
🤖 Bot: Welcome to my website!

👤 You: Are dogs cute?
🤖 Bot: Dogs are cute.

👤 You: And cats?
🤖 Bot: Cats are cute.

👤 You: What about cows?
🤖 Bot: I do not know.

👤 You: why?
🤖 Bot: You might ask me why cows are cute.

👤 You: Yes but u dont know?
🤖 Bot: If you have cows, you should know.

👤 You: ```

its.....

idk

[serene scaffold]

I kinda like it tbh

Do you like this cat?

[orchid light]

yes

And btw do u have some datasets that i could try to train it on?

[serene scaffold]

No

[orchid light]

🥹

[serene scaffold]

Yeah sorry
I just literally don't, lol

[orchid light]

Its ok

I will try to fix it

and then if i get it working it will be my base for a vtuber (ofc it will be alpha v0.1)

i plan on training an MoE model next

[agile cobalt]

if you haven't yet, try talking with SmolLM, Gemma and Qwen models of similar size to see how well they work

imo under 3B it is hard for the model to have a concise conversation

[orchid light]

Yeah and i think its still a problem of finetuning....

[opaque condor]

What book is simple for AI because I have the pie torch manual it's like a reading a language that I don't understand I've made some networks but that's from YouTube videos I want to have a book in case YouTube takes down those videos

[opaque condor]

Everything there was a book on the craft course they put a video or a link I can't find it check the PIN wasn't there

[mellow vector]

anyone have some good resources discussing when or when not to normalize?

[spring field]

I suppose if the data is already normalized

[worldly dawn]

at scale or in some nosql context, you might denormalize it

[fierce creek]

for a beginner llm (running on an m4 chip and integrated graphics), how much data should I use to get somewhat of a decent model? I'm not really going for anything crazy, just to learn. also how much parameters should I use?

maybe when using tree based models

[desert oar]

Are you talking about normalization of features, standardization of features, or database normalization?

Where normalization means rescale to a fixed interval such as [-1,1]. And standardization means subtract the sample average and divide by the sample standard deviation

[mellow vector]

I'm going over batch norm atm, looking around the net some dispute over it's use came up but it sounds like for the time being I can probably get away with normalizing everything.

[desert oar]

Yeah so for the future, it helps to clarify what exactly you mean by "normalization". You got 3 different answers and none of them are relevant to the question you asked

[mellow vector]

sort of covers it, I was interested in more on normalization than just batch norm

[desert oar]

You might not have realized it but it's one of the most overloaded terms in statistics and math, meaning that it has been unrelated or loosely related meanings

[desert oar]

I see. Maybe a very general answer is that normalization constrains the scale of the numbers you're working with

That's true for statistical standardization, vector normalization, or normalization" meaning rescaling min/max to a defined range

And, in general, that tends to help with numerical stability

But it's also really important whenever you're working with combining data of various scales. You need to prevent the data on a larger numerical scale from overwhelming the data on a smaller scale

[stray igloo]

Good night for everyone 😄
To give some context on the following question, I'm a Data Science Engineering student with high interest in ML so I was thinking to do a certification a side of the career I'm studying rn. So, should I go for AWS, Azure or Databricks certifications? if any of you guys have any other suggestions I would appreciate it.

[pulsar mural]

@stray igloo can you explain what are the things to study for data science, what is the main work as a data science engineer ? Just you filter the efficient data from the raw data by applying data science skills ?

[stray igloo]

Na, actually is the same as data science in other places but in my university they wanted to add some engineering classes

[hollow fossil]

im training a CNN is it advisable to have a batchnormalization layer after each Convulutional layer ?

[stray igloo]

It depends on how your architecture is but most of the time, it is

[rotund walrus]

ahh yes new month, new quota

[rich river]

color_img = self._cv_bridge.imgmsg_to_cv2(
    img_msg=msg.color_image, desired_encoding="8UC3"
)
...
class YoloModel():
    def __init__(self, model_path: str, confidence: float, max_depth: float):
        self._model = YOLO(model_path)
        self._confidence = confidence

    def set_color_img(self, color_img: np.ndarray):
        color_img = cv2.cvtColor(color_img, cv2.COLOR_BGR2RGB)
        self._color_img = color_img

    def segmentation(self) -> Tuple[list[str], list[Segmentation]]:
        model_output = self._model.predict(
            self._color_img, conf=self._confidence, iou=0.65, agnostic_nms=True
        )[0]
    ...

I found that the line color_img = cv2.cvtColor(color_img, cv2.COLOR_BGR2RGB) is required here to get the output
but sometimes it seems it is not required?
I wonder when this line is needed

[tacit basin]

Looks like it's converting BGR to RGB. So I would say that it's not needed if it's already RGB

[iron scarab]

Sry my question not related to the topic but where is free topic discussion?

[mild dirge]

opencv reads images as BGR by default, even if it is a regular .png or .jpg :/

[cedar tusk]

free as in python general or off topic completely

[opaque condor]

how many .txt files do i need for a gpt modle

[waxen kindle]

that's not really relevant, you can have 1 file with billions of lines and billions of files with 1 line

[flint ore]

Why would you have billions of files with one line

super inefficient imo

[lyric fern]

Hi guys , I am new to this community and python , currently I don't know much about python

Can anyone suggest me a course or a video from which I can learn it ?

[waxen kindle]

of course, but that's only to show the irrelevance of the question

[waxen kindle]

!res

[arctic wedge]

Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

[waxen kindle]

we have a resource page

[lyric fern]

Ohh

[opaque condor]

I'm sorry for asking I know i it seems real simple and stupid I want to train my own mouth from scratch

[waxen kindle]

it's not stupid or anything, the thing is that the number of files doesn't matter, what matters is their content

also good luck to train a model from scratch, It would be easier to fine tune an already existing one

[opaque condor]

So I gave it an entire manuscript of human evolution of human history human art all in the same file with that be okay

[opaque condor]

As an example or do I need to do something different

[waxen kindle]

the important thing is how you handle the files

[waxen kindle]

training a good gpt is expensive, idk what you can achieve with a PC

do you have any idea on how to proceed ?

[opaque condor]

Yes

[waxen kindle]

and do you have the resources to do it ?

[lyric fern]

Do you guys making a gpt model ?

[waxen kindle]

uhm, Mechanical fox is, as you can read

[serene scaffold]

only large companies with exceptional resources can make a GPT model.

[lyric fern]

I know I was just asking , I read somewhere it need billions of dollars to make

[waxen kindle]

it does

[lyric fern]

I was just wondering if we can make a J.a.r.v.i.s model not exact same like movie because technology is not that high but close to it

Is it possible?

[jaunty helm]

no
the current best and/or most expensive models on the market can and do still fumble on seemingly simple tasks, definitely no where near the AI you might be used to in scifi movies

[lyric fern]

Hmm thanks for clearing my doubt

[opaque condor]

how many words do i need for each file

[past meteor]

Strange there's no gpt 4.1 in there

4.1 is so good

I don't know anthropic's lineup well enough, but they have no long context models iirc

Does the LLM you're using make a noticeable difference (aside from "big" jumps like reasoning vs no reasoning or tool calls)?

Hot take but I feel like somewhere in the past 6 months the average model got "good enough" for the average task

[lavish wraith]

Is data engineering and software engineering is same ??

[past meteor]

Not at all

[serene scaffold]

data engineering is when you develop the data infrastructure for an organization.
software engineering is general.

[past meteor]

They both write code, but very different kinds of code

Lots of data engineers don't know (a lot of) Python and do the vast majority of their work in SQL

And many software engineers don't know (a lot of) SQL

Depends on how much is in your prompt indeed

At some point of adding details 4x models just give up

The place where I see LLMs still struggle is numerical stuff ofc

[lavish wraith]

Ok thanks for reply is data engineering job is a hype or fake ?? Could data engineering use in AI applications ??

[past meteor]

Neither hype nor fake, actually the "inverse", it's a very conserative job/career path at this point.

Honestly, I think AI applications will increase the need for some sort of data engineering. I think low-code or SaaS AI apps make the AI side of things a commodity so the challenge going forward is probably going to be centralizing and curating (meta)data

[lavish wraith]

I read IBM article on here it said that data engineering is software engineering

"Data engineers are software engineers who build and maintain an enterprise’s data infrastructure—automating data integration, creating efficient data storage models and enhancing data quality via pipeline observability. Data scientists and analysts rely on data engineers to provide them with the reliable, high-quality data they need for their work "

[past meteor]

From the outside looking you can classify it as software engineers

But to people in the know - no

[lavish wraith]

Ok

[indigo bear]

hi does anyone know any good courses on making AI's on nash equilibrium stuff like for poker and other stuff?

just basically on Imperfect information games

thanks

[past meteor]

Why do you update the system prompt instead of moving some of this stuff into context

[shy marsh]

hiii, im starting as data science :>>

[peak junco]

Hello everyone. I have a question regarding AI agents. Are agents capable of executing tool calls themselves, or only capable of emitting tool call events?

[peak junco]

Thank you very much. In summary, agents cannot execute tools themselves.

Not exactly. The agents return a structured response along with their original response. The structured response contains the function to be called, along with the necessary arguments. So while the agents themselves can't execute the tools, their response can be parsed externally by a program which can then execute the function along with the provided arguments.

[peak junco]

I wouldn't say the program external to the LLM that executes the code is part of the agent. Atleast in my case, since I'm working with Azure AI Agents. But yeah technically if you're building an agent as part of your software and not invoking a pre-deployed agent, then yes I agree, that's a good definition of an agent.

[peak junco]

I don't think so, one core part of agents is that they support state management. But I agree with the second part. Use it as a reasoning agent for task orchestration.

[agile cobalt]

For most practical purposes, you'll want to consider an Agent to be a combination of the LLM, the tools it has access to, and some configuration parameters such as its system prompt and temperature

What you're defining there - handling text inputs generating text outputs that must be parsed by something else - is only the LLM part

It is obvious that the language model itself is not able to execute code directly, but a lot of frameworks go as far as abstracting tool execution such that it happens under the hood (i.e. you don't have to call it yourself, only give the agent its tools then the framework runs them for you)

[peak junco]

Yes, completely agree with that.

[peak junco]

Really? Wasn't aware of that. Maybe because Azure agents are different from other agents. Because Azure agents are incapable of executing the tools themselves.

[agile cobalt]

Taking a look at their docs, looks like they try to give you as much control over custom tools as possible, but they have some built-in tools that they do execute themselves (not the LLM, but Azure's service)

the MCP server support seems to be a bit more "automatic" though, someone still needs to run the MCP server in first place, but looks like you can just connect and let Azure call it automatically

• https://learn.microsoft.com/en-us/azure/ai-foundry/agents/how-to/tools/model-context-protocol
• https://learn.microsoft.com/en-us/azure/ai-foundry/agents/how-to/tools/model-context-protocol-samples

also there exists an official Azure AI Foundry discord server you might want to look into

[peak junco]

Wow that's interesting. You can connect your agent with other Azure services. I won't be using it currently since the tool interacts with Snowflake and not any Azure services, but still, that's interesting. Thank you for these references as well. Learned many new stuff today.

[peak junco]

This is kinda similar to what I'm doing, except I only have 1 tool to be executed. Based on future requirements, if the system calls for multiple tools, I'll definitely be going with this. Also, sorry if this is a stupid question but why are agents much much slower than models? Agent response takes 3 times the duration invoking a model takes.

[worthy vigil]

Hello, I'm currently facing a Data-Science Problem where I have a large number of 3D points (each with a timestamp). One point stands for an event which happens at the points location.

What I want to do with this data is to detect local changes in the number of events at a certain time interval. (e.g. is there an area where the events at some time interval rapidly increase/decrease).

My first thought was to do a KDE (Kernel Density Estimation) for all points within a certain time interval and compare the densities of different time intervals with each other or to cluster points in space and time.

Maybe someone here faced a similar problem and can give me some hints what I could try or can hint me some further resources.

[sly isle]

Are you guys excited for this book coming in December this year?

[opaque condor]

Is there a person's multimodal that can both generate music/audio ,text, images,video at the same time

[calm cipher]

ooh a new edition? what changed

oh it's PyTorch not TensorFlow

oh dang that's awesome, if I were still teaching undergrad ML I'd switch to that in a heartbeat

[rich river]

not really, I wonder how important sklearn and traditional ml is in today's LLM world

[jaunty helm]

very
not all (in fact, a good chunk of) problems can't and shouldn't be "solved" by llms

[mellow vector]

I've only just scratched the surface of ML and I'm not familiar with any of the books, but every time I go over a new subject I'm tempted to peek under the hood and try my own customizations with sklearn.

[tacit basin]

It depends

[rich river]

llm dominates

[jaunty helm]

dont agree
simple example; would you trust a weather forecast by an llm or literally any of the other tried and true models

not to mention useful llms use billions of parameters

[waxen kindle]

No, there are tons of applications in which you don't want to use LLM, actually, most of the time you don't

[peak junco]

Omg yeah, I had to learn a lot about how agents work. It makes tons of api calls for each query because of state management and thread management. Api calls for tool execution and returning the results as well. Gave me a headache trying to wrap my head around how it works internally.

[robust granite]

anyone here who have some expertise on databricks?

[serene scaffold]

Hello, please remember to always ask your actual question. Never ask to ask ever.

[cursive totem]

I wonder how do you have different profile picture in chat but always the same when i click on you?

[agile cobalt]

nitro users can change their avatar for each server they are in (Server Profiles), but depending on where you click it may show their Main profile instead

[abstract wasp]

Wait are the leetcodes the regular leetcodes or are they like ml based?

[lyric fern]

Broooo , I also want to make a jarvis like it but on a website not dc , can you help me on it I am new on programming

[abstract wasp]

Wow really cool which technologies did u use?

Ooo ok ty

[lyric fern]

I know , I am just excited to build my jarvis not like same in movie because tech is not that high enough but like he can do most of the things

[abstract wasp]

Im currently also building a rag llm w langchain and I was thinking of making it agentic too

Really cool

[lyric fern]

I tried gemini api on my first project and also added pyttsx but it didnot worked , it showed errors like wrong api key , quota exceeded..

Even though I created new api key and chat gpt told me it's free and unlimited so how my limit exceeded?

In how much time you learned python like , if I learn 1 hours daily so on how much time I can become intermediate at python ??

[waxen kindle]

that depends on how fast you understand concepts

[abstract wasp]

Im still not sure what sort of things I’d want the agent to do for me but prolly nothing too complex

I really want to do a masters too but I’m not sure in what, more ml or a lil diff like electrical eng ;-;

[peak junco]

If you have a powerful PC, and if you're just learning, you can download and use models locally using LM Studio. No request limit. If not, you can try using Cloudflare Workers AI. Offers a higher limit than other APIs.

[opaque condor]

Has anyone ever used panda3d for AI training

[iron basalt]

Yes.

[opaque condor]

How'd it go

[iron basalt]

Works fine. Panda3D is a very solid game engine.

And since it's as a Python library, it fits nicely.

[opaque condor]

What did your teach your AI?

[iron basalt]

Virtually evolved creatures.

[opaque condor]

Genetic algorithm?

[iron basalt]

Yeah.

[opaque condor]

What current batch are you on

Or genetic creature?

[iron basalt]

It's more for quick from scratch to test algorithms. I have not left it super long running since I don't need to.

[opaque condor]

Why not try running it for as long as you can see what the program creates for evolution sea what it creates if it's running for a long time

[cursive totem]

Is antimalware on pytorch rare problem?

[iron basalt]

Because i'm developing algorithms and it's usually immediately obvious it won't work. So lots of fast iteration to try things.

Long running not yet.

I have done that with others, just not P3D.

P3D is just way easier to mess around with than the common distributed simulators.

[opaque condor]

What about adding variable gravity so when you use your simulation date teacher AI you can teach it to jump from and to celestial bodies and account for its gravity and entry orbit etc

[iron basalt]

There are many things i'm testing, that is not one of them but highest jump is one.

[opaque condor]

My plan is to make an AI similar to Kane from the digital circus and then make a game for glitch

Also do you know how to program your own environment

[serene scaffold]

Squiggle knows how to do everything

[iron basalt]

Game development and physics simulation developement experience. https://karlsims.com/evolved-virtual-creatures.html

For an idea of what i'm doing.

[opaque condor]

Copying the link for later because I should probably start working on designing an environment

So you've made an AI that walks too

[opaque condor]

That's cool

Also in general has anyone taken in AI like a language model and give it a 3D environment to work with

[iron basalt]

Yes.

[opaque condor]

How was its performance

Squiggle

[iron basalt]

The performance is more than good enough.

The heavy work is offloaded to C++, just like how one normally does in Python with most of its libraries.

[opaque condor]

Have you tried it with a language model that can produce images sound and everything relative to image generation audio generation etc and how long did it take

[opaque condor]

How can I get the AI be able to use it joins do I have to put specific bones and specific areas in blender to get it to move

[safe agate]

Sasha Rush created these Tensor puzzles to help people learn PyTorch. They're fun and interactive with immediate feedback from cute dogs.
Try to solve them and let me know how many dogs you find! https://molab.marimo.io/notebooks/nb_5x6yE9oNBgU164M1DPHZS5

[kind jay]

How can I make Ai I am in a competition so any one pls help to make the best

[burnt sedge]

Question about sympy
In physic you can have Voltage that have 3 different formulas for calculating it

V = IR = I/P ​= sqrt(PR)
​
How to make in sympy a variable called Voltage that could have 3 different equations?

I = Symbol("I")
R = Symbol("R")
P = Symbol("P")

Voltage = Eq(I*R, I/R)

# But you can't make
Voltage = Eq(I*R, I/R, sqrt(P*R))

[waxen kindle]

I feel like it's not possible, based on how sympy works. But anyway, you can link P to I and R so at the end, maybe with 2 equations it will work.

I = Symbol("I")
R = Symbol("R")

P = Eq(I*R*R)

Voltage = Eq(I*R, I/R)
``` (or another equation based on which of the 3 you know)

[serene scaffold]

you can ask about sympy in this channel.

the name of this channel was originally just "data science" and we added "ai" to make it more clear what the intended scope is, not to actually increase the scope.
the description of this channel is "For discussion of scientific python, matplotlib, statistics, machine learning and related topics."

[waxen kindle]

Alright yeah I missread the "scientific python" part, my bad, I'll delete everything

[burnt sedge]

Finally solved it. Okay. If someone interested, here is and explanation with simpler example. And you can add different class methods for representing type of formula, but need to be extended in __new__ too

from sympy import Expr, Symbol, pi

class Circumference(Expr):
    r = Symbol('r')
    d = Symbol('d')

    def __new__(cls, radius=None, diameter=None):
        if radius is not None:
            return 2 * pi * radius
        elif diameter is not None:
            return pi * diameter
        else:
            raise ValueError("Provide either radius or diameter")

    @classmethod
    def from_radius(cls, radius):
        return cls(radius=radius)

    @classmethod
    def from_diameter(cls, diameter):
        return cls(diameter=diameter)

C1 = Circumference.from_radius(Circumference.r)
print("Circumference from radius r:", C1)  # 2*pi*r

C2 = Circumference.from_diameter(Circumference.d)
print("Circumference from diameter d:", C2)  # pi*d

[waxen kindle]

so that's basically creating an expression based on what you know and not creating one that could have mulitple equations

[burnt sedge]

Yes. With Eq you can select type using subs with intended symbols to substitute. But you can't have more than 2 expression for one variable, because I think there could be collisions. So using class like this derived from Expr I think much readable when creating an instance

[safe merlin]

Whats the best way to install panda?

I would say im a beginner

But im following this tutorial right

https://www.kaggle.com/code/dansbecker/your-first-machine-learning-model

And I cant seem to get the data right

no like

FileNotFoundError: [Errno 2] No such file or directory: '../melbourne-housing-snapshot/melb_data.csv'

import pandas as pd

# save filepath to variable for easier access
melbourne_file_path = '../melbourne-housing-snapshot/melb_data.csv'
# read the data and store data in DataFrame titled melbourne_data
melbourne_data = pd.read_csv(melbourne_file_path) 
# print a summary of the data in Melbourne data
melbourne_data.describe()

Thats my code in vs

?

like downloads

Well i dont get any error

but i get no output

what abt sklearn?

yeah it was i installed it

[sterile cypress]

Guys I am spidy the new joinee

And I am a beginner coder in python that's all about me I am here to seek some help to improve my skills

[merry orbit]

So to make a long story short I'm trying to extract the calendar from picture 1 to the calendar I made in picture 2, right now I have tried to use easyOCR and then using a Ollama model to remove duplicates and fix word in that language in case there are any, but I have the problem where sometimes easyOCR returns a list of words with strings that get connected so some things like certain things under dates get connected into one string, pytesseract is even worse at this when I tried to use instead of easyOCR and right now I'm kind of stuck because I'm trying to add the .png file of a schools calendar into the other calendar, so if anybody has any other solution or suggestion it would be helpful.

um

can you speak in peasant terms

cause I didnt understand any context from that sentence

I mean I had no idea you could upload images to those models with python, I used chat-gpt api before, does gemini have an option to add images using a path in python or?

Yeah I actually love that idea

[rare coyote]

hi guys. im a 14 year old interested in learning AI/ML and eventually DL. Any suggestions on where to start?

[merry orbit]

damn dude you saved me

now I dont even need an uneccesary ollama prompt

with gemini it works perfectly

thanks

[mellow vector]

anyone play with polars.DataFrame.to_torch? should I stay away?

[serene scaffold]

why do you want to stay away?

[mellow vector]

it's currently experimental

just considering memory optimal pipelines but it's really more a curiosity than anything

[mellow vector]

spent the week incorporating duckdb, marimo and polars into my stack

[agile cobalt]

it is fine to use it, they keep many things as experimental because they don't want to bump up the major version anytime soon & those methods might be redesigned, but it is rather unlikely for it to be outright removed

if you weren't supposed to use it, it would either not have been released at all or be private/undocumented

[hollow fossil]

mobile = keras.applications.mobilenet.MobileNet()
x = mobile.layers[-6].output
output = Dense(units=10, activation='softmax')(x)

i saw this code in a tutorial what does this do exactly ?

[waxen kindle]

MobileNet() is a neuron network

The second line get the output of a layers

The last one compute probabilities to classify the input

It's a little weird written as it is, bc the last layer neeeds to be trained

And no input is given to the model

But that's the idea

[eternal turtle]

Hi! I'm currently learning ML and PyTorch and have some experience with fullstack development. I was wondering if there are any of your own interesting GitHub projects you guys are currently working on that could use some help. I’d love to open PRs and learn more about deep learning in practice.

[next shard]

Hii guys

[next shard]

I'm making a website named as data insight pro

[eternal turtle]

nice. just send me the link privately if you want and i'll look into it and tell you if i could help you out on it ( maybe rate your idea or suggest things😆 )

[next shard]

Umm it is on my GitHub but it is not up to date

[crimson jackal]

In a monte carlo algorithm, if i want some variables to be positive, the best way to get them is to use a step function that rejects negative values? Even if the they are a function of a high number of variables?

If I use a soft approximation wont it make the values be positive but possibly close to zero?

[sly isle]

Why are there so many open issues? Won't anyone close them? 🤔

[sly isle]

I see the reason behind that. However, you can probably spend some time to check issues that are open for more than 15 years, right?

Is it possible to integrate an open source library to The Python Standard Library?

Do you know uv? 🤔

No, it's not.

It's well-known that uv is a game changer for today's package management in Python. Why wouldn't the Python Software Foundation integrate such a beauty in the The Python Standard Library? 🤔

[agile cobalt]

https://peps.python.org/pep-0002/ describes the process for it, but it is extremely rare

After a module enters the standard library, it cannot be updated as often, it's essentially frozen in time. Some people even say that "The standard library is where modules go to die."
even pip is, strictly speaking, not part of the standard library

It also is becomes the responsibility of Python’s development team, not of its original developers. uv is managed by Astral, which intends to integrate it with their commercial offerings in the future

The idea of adding it to the standard library gets even worse when you consider that despite being a tool made for Python users, uv is mostly writen in Rust

[sly isle]

Ah, I see. That sounds terrible. Besides that, will Astral really charge us for using uv? 😟

[agile cobalt]

no, but they will offer services that uv natively integrates with at some point

similar reasons for numpy btw
even ten years ago:

The standard library has a slow release schedule, maintains a very high-degree of backwards compatibility, and tries to greatly limit external dependencies. Numpy has a much faster release schedule, has a bunch of large external dependencies, and although it has a high degree of backwards compatibility it is not as backwards-compatible as the standard library and some bigger backwards-compatibility breaks are being discussed.
(arguably + keeping it the standard library lighter)
and the reasons stay mostly the same today

[crimson jackal]

They are values for masses so they must be postive.

[serene scaffold]

making it part of the standard library would just make things more challenging for no added benefit. it's trivial to install uv, and making it part of the stdlib would make uv's development cycle coupled with cpython's development cycle.

[crimson jackal]

I dont think I can.

There is not a direct correlation between the distribution I am using and these specifc parameters.

Well, i dont think there is. But I dont see how changing the distribution will change that since I dont have any way to know due to using a large number of variables.

My initial conditions are very broad.

Okok let me try.

Ok, first of all since each step is a random change in all variables you can enter a non-physical point and when that happens the likelihood goes over the roof. So its normal to get negative masses.

Since this is a monte carlo algorithm you have random points.

Fair enough. What I am trying to solve is to find the values of my model parameters such that when computing some values, there is an agreement with the numerical and known experimental results. I am trying to not enter much on the physics since its a heavy topic.

[dense surge]

Hello. I've been struggling with a certain data-science-related problem for a long while now, and after giving up a few times, I have finally decided to try reaching out, to see if there's anyone smarter than me that would be able to figure it out. I've put the entire thing into a stackexchange post here: https://datascience.stackexchange.com/questions/134408/product-distribution-between-a-set-of-machines

Every time it feels like I can grasp how to solve it, it keeps turning out to not be as easy as it seems. As the comments under the single present answer show too.

[desert oar]

At first glance, this looks like an operations research task

Ah I see you found OR tools

It looks like you were given a very good starting point

And the challenge I think is that it can sometimes be very difficult to express a real world problem in a way that you can solve it without inventing new math, or resorting to something very esoteric

I'm not sure how to jam this into a linear programming framework because OR and optimization techniques are not something I have studied in detail. But maybe that uniformity constraint can be expressed with a penalty term on the variance among machine utilizations

[serene scaffold]

@desert oar welcome back

[fickle shale]

yeah it's an or problem!!

may be look like job scheduling problem!!

Better to ask on operational research server!!

[dense surge]

Hmm. Is there a server like that?

I did some very extensive googling, and couldn't come up with a description on the kind of the problem even, so I finally tried asking ChatGPT, and it pointed out it's probably a linear optimization problem. I found OR Tools from there.

The only response under the post so far does appear to work, but not in all cases, so the solution is incomplete =/

I found some kind of operational research server off google, but it's only like 300 people big. Not sure if that's the one.

[umbral star]

@pine wolf meet @desert oar

(I thought both are the same ppl :p)

@pine wolf batgrl is quite too good wtf

[past meteor]

If you’re unsure about this stuff you can typically reformulate it and use metaheuristics to solve the problem

Which is cutting a tomato with a chainsaw if it’s a LP or IP, but I find them easier to formulate if I’m unsure of stuff

[desert oar]

There is an OR stackexchange site. You can post a followup question there

[desert oar]

What are metaheuristics?

[exotic star]

guys while learning linear algebra is there anything i can do to implement it in python and ML? so i dont learn dry

also this year i have linear algebra, analytical geometry and algebra in school but probably the basics only so ig it could help

maybe i could do some numpy projects, if u have any ideas it would really help

[desert oar]

Implement what exactly? lots of algorithms on matrices and vectors are already available in numpy and scipy. also in many cases the details of implementing algorithms from linear algebra can be complicated, and it ends up being kind of a separate topic from learning the linear algebra itself

If you want to work through calculations to show how a particular theorem works, or to check your work, then numpy and scipy should help

[past meteor]

Genetic algorithms, tabu search, simulated annealing, …

[restive storm]

Could anyone suggest a good course to learn data science.

For free

[serene scaffold]

!resources data science

[arctic wedge]

Resources

The Resources page on our website contains a list of hand-selected learning resources that we regularly recommend to both beginners and experts.

[sterile cypress]

Guys in this group will they send any hackathon event

Like any event

[exotic star]

yea just use numpy to work trough linear alg concepts/problems...and see how they work in practice

[grand minnow]

You can check the #announcements . The last event they had was a Code Jam #announcements message I'm pretty sure that there will be more events coming up

[bleak dew]

I guess this is the channel to ask pandas questions. I've created a full outer merge with separate key columns for a and b. However I'd like to create a joined key column c that contains the union of the two key columns. How can I do that in a simple way?

I wonder, does df["c"] = df["a"].fillna(df["b"]) work?

[cinder schooner]

Hi everyone,

I recently wrote my first blog post where I share a method I’ve been using to get good results on a fine-grained classification benchmark. This is something I’ve worked on for a while and wanted to put my thoughts together in an article.

I’m sharing it here not as a promo but because I’m genuinely looking to improve my writing and make sure my explanations are clear and useful. If you have a few minutes to read and share your thoughts (on structure, clarity, tone, level of detail, or anything else), I’d really appreciate it.

Here’s the link: https://towardsdatascience.com/a-refined-training-recipe-for-fine-grained-visual-classification/

Thanks a lot for your time and feedback!

[serene scaffold]

this means "define new column c that is a copy of a, but where missing values in a are the associated value in b"
this means that a takes priority. if both a and b are non-null, the c value will be the a value

[desert oar]

This is a really nice rundown of deep learning optimization techniques. Helpful for someone like me who is pretty far out of the loop and would have a lot of work to do in catching up without resources like this

It's also a nice write up, easy to read and it also talks about why you made each decision which is nice

I said "nice" 3 times so that is how I feel

That's it, maybe this is a silly question, but did you consider any kind of hierarchical or two stage prediction to take advantage of the idea that two cars of different models but the same brand, or different model years are the same model, might be very similar?

I worked on a very big project that included a modeling component like this, where you had a few dozen broad categories, but each category had a few dozen smaller categories inside it. And the ability to distinguish small categories varied a lot depending on the broad category

I admittedly did not click all of the links to the research literature you referenced l. so maybe that's in there, and you didn't pursue it because you wanted to try the simpler approach first and it worked well enough that you didn't need it

[cinder schooner]

Thank you very much for the feedback, yes i did but this is only the baseline on which i can compare everything that i did after. Like to know if the solution i proposed works i would compare it to the results i get here. What i did was i added multitask learning with the new tasks being predict the car company and the segment of the car ( is an suv, a berlin, a compact, a 4x4 etc)

[desert oar]

By that you mean, you have multiple outputs in the same model?

[cinder schooner]

I changed the head so it gives more classes and then i compare the first part to the finegrained classes, the second to the company the third to the segment and then the loss is calculated as w1 * loss 1 + w2 * loss 2 + w3 * loss 3, the weights (w1, w2, w3 being hyperparameters)

[viscid fiber]

Good afternoon everyone!
I'm trying to write a code which performs many operations on a pandas dataframe containing financial data.
As the dataframe has thousands of rows, I'm a bit worried about performance.
Thus, I'm trying to "time" every operation.
However, the results are the opposite of what I would expect from common knowledge and common advice
(for example: https://stackoverflow.com/questions/16476924/how-can-i-iterate-over-rows-in-a-pandas-dataframe/55557758#55557758)

I'll add a few details for context

For example, one of the first operations I need to perform is finding the length of the code in one of the first columns
(the column contains codes which are comprised of numbers separeted by dots, like: 1, 1.1, 1.1.1, 1.1.2, etc.)

This is the column with all the codes after I split them

Common sense dictates that for loops in Python should be really slow compared to other methods

However, the for loop is more than twice as fast then vectorization and apply()

[wintry relic]

that's not the for loop's fault (being slow), actually, that's the generator's

[agile cobalt]

.apply() is just as bad as for loops

usually vectorization (.str.len()) should be faster, but operations with strings are pretty slow with pandas overall

"thousands of rows" is extremely small though, even if your code is extremely inefficient it'll still be fast until you get at least x10~x100 times larger than that

either way I'd recommend taking a look at polars, it is another dataframe library similar to pandas but more modern, faster and (arguably) has a better API

[viscid fiber]

Ok so I'll just choose the option which is the most readable instead of trying to find the most efficient

[serene scaffold]

I wonder if .str. operations are faster with the pyarrow backend (since the strs aren't python objects, in that case)

[viscid fiber]

the .str. operation is faster when applied to actual strings instead of lists

I actually can't even understand how .str works on lists anyway

[agile cobalt]

seems like it uses this: https://github.com/pandas-dev/pandas/blob/567de07bbcfa07e3bb8945b1193cab40cc6be892/pandas/_libs/lib.pyx#L2951
it uses cython to speed things up a little bit, but is still kinda slow given you're dealing with python objects, at least compared to native rust/C performance you can get with other libraries

[arctic wedge]

pandas/_libs/lib.pyx line 2951

def map_infer_mask(```

[agile cobalt]

from some quick testing, doesn't seems like it is much faster (if at all)
not sure if I setup something wrong though

(polars is faster though)

[glacial sable]

what is this?

[desert oar]

What do you mean by "first part"? Like you concatenated the one-hot encodings of classes at all 3 levels?

[desert oar]

The default string dtype backend is numpy fwiw, unless you mean object style

[desert oar]

It's not operating on lists, pandas .str accessor methods basically do internally what you did manually with Series and list comprehension

That is, lists do not have that .str attribute. Only Pandas series have that

[gleaming lake]

I wanna be an ml engineer cool dream right? Well if you're a 13 yr old overthinker not all that good

I litreally go into overthinking sometimes depressed states after seeing how a neural network looks likes although I'm learning python bit by bit this does demotivate me a little and i try to think will i be able to do it or not please help me so I can stop overthinking

[lapis sequoia]

Hi

[cinder schooner]

I will send you a graphic tomorrow, its basic multitasklearning

[crimson jackal]

Hello. I have a problem where I need to determine 28 variables such that they predict 6 physical constants. I am using markov chain monte carlo method to try to do it with random increments to my initial condition, but since my initial conditions are very broad and there are some conditions that each point (the 28 variables) must met or instead the likelihood fucntion returns inf. I did try to find some alternaltives to this method since its being very dificult to get a good result or for all conditions to be met but I lack some statistic knowledge to actually understand if the other methods are better or not. Is there a better method than this one?

[opaque condor]

Can someone help me with figuring how to agment data

[calm cipher]

you can just ask your question, if anyone knows they'll reply

[calm cipher]

Neural networks are a lot to get into and build on a lot of math you probably haven't been exposed to before, including calculus and linear algebra, it's OK to be intimidated by them

Have you looked into other methods of machine learning? Linear regression is much more intuitive even if it still uses some advanced math, and actually you can think of neural networks as an extension of linear regression

I guess I'm saying if this is your introduction to machine learning then you're starting with the big stuff, make sure you know the building blocks first and it will be much less intimidating

[wicked lance]

what just happened with the se4rverA?

[opaque condor]

how many images do I need for a costomedata set for a neral network

i currently 33 images

all labled in one of the sub image folder

[desert oar]

You have some nonlinear physical model already that you want to understand the behavior of, or something else? Where does a likelihood come into play here?

[crimson jackal]

The lkelihood will tell you how well the physical model fit the physical constants. Like you need the mass of a variable A that is computed with the input parameters, to be equal to, lets say, 10g. Here the likelihood will tell you how close the model gets 10g for variable A.

[crimson jackal]

I have physical model where the parameters that describe the model must be determined. The model must obey experimental data, and so we want that the model with the respective parameters are able to obtain values equal to the experimental data. The likelihood come into play here as esplained in the above mesage.

[desert oar]

So this sounds like a pretty typical statistical model fitting task then?

You have 6 features of experimental data, and a probability model that describes the joint distribution of that data, and 28 parameters in that probability model?

Or do you mean something else by "likelihood"?

[crimson jackal]

My likelihood is how well the input (28 parameters) fit the 6 features of experimental data. That is with the input i compute the respective value associated with the experimental data and with a chi2 test I see how close it to the experimental data.

[desert oar]

OK, I would avoid using the term "likelihood" because that has a very specific meaning in statistics and probability, and it doesn't sound like that's the meaning you have in mind

[crimson jackal]

Oh ok.

Sorry I am a little behind in statistics.

[desert oar]

What do you mean by this?

with the input I compute the respective value associated with the experimental data

what is the input? What is the respective value?

It would help to be a little more precise in your explanation of what you are doing, then you're more likely to get a helpful answer or suggestion

[crimson jackal]

So, the input are the 28 parameters. With this, for example, i compute a hessian matrix and get the eigenvalues. The first eigenvalue is one of the values that must fit with one of the experimental data.

[crimson jackal]

Okok sorry.

[desert oar]

OK, I think we are getting closer to a useful explanation

So you have some function, maybe something derived from theory in your field of study, that has 28 parameters as input, and 6 numbers as output

And you performed some kind of experiment measuring those same 6 numbers

And now you want to find the parameters that best fit the data

Is that right?

[crimson jackal]

Yea.

[desert oar]

and it sounds like that function can be quite complicated, for example it involves the eigenvalues of the hessian matrix of some other thing

[crimson jackal]

Yea, its even more complicated.

It has some conditions too. Like all the eigenvalues must be positive to be physically accurate.

[desert oar]

So the parameters must be chosen in order to maintain those conditions?

And your approach so far has been to choose initial values for the 28 parameters, check if they are valid, and then randomly perturb the parameters to see if the goodness of fit improves with respect to the experimental data? Discarding any parameter combinations that result in violations of theoretical constraints

[crimson jackal]

Yea.

It does not seems a bad method but with all these constraints its been kind of hard to get good resutls.

Even more the code is not finding values that actually obey all the constraints.

I dont think the model is wrong but it can be, but for now I would like to think and try other methods to try get something.

[desert oar]

I think I have an idea for what is essentially a more principled version of the technique you came up with, which might help because there are software libraries that implement it

But, before I suggest it. Can you describe in more detail how you are measuring goodness of fit?

You mentioned a chi square test of some kind

[crimson jackal]

Yea.

Essencially its (numerical - experimental)^2/ standard deviation^2

[crimson jackal]

I am using the mcmc with emcee. If you know them.

[desert oar]

standard deviation of what in this case?

[crimson jackal]

Of the experimental value.

[desert oar]

Yeah that's my next question. What you described doesn't sound like much of an MCMC procedure. Monte Carlo perhaps, but not MCMC as such

Not sure if that's terminology confusion, or I'm missing more details

Normally MCMC refers to a specific procedure where you construct a Markov chain such that the stationary distribution is a particular probability distribution that you are trying to sample from. For example the Metropolis Hastings algorithm

And that's why I got confused when you used the term "likelihood", because MCMC is often used in a context where likelihood is very important as well

[crimson jackal]

Damn, my lack of knowledge.... ahahha but I think is that.

Sorry I am starting to confuse myself...

So, I am using emcee and mcmc from emcee. But I thought what I said earlier was correct even in this context.

[desert oar]

If your code isn't too long, maybe sharing that would help clarify

Because yes emcee does MCMC

[crimson jackal]

Ok, the error function is a little too long but the other one is short.

def run_MCMC(p0, lnprob, ndim, nwalkers, nburnin, nrun, h5name, scale, reset=True, pool_obj=None, burn_in=True):
    
    initial = [np.array(p0) + 1e-4 * np.random.randn(ndim) for i in range(nwalkers)]         

    backend = emcee.backends.HDFBackend(h5name)
    if reset:
        backend.reset(nwalkers, ndim)

    sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, backend=backend, pool=pool_obj, a = scale)

    if burn_in:
        print("Running burn-in...")
        print(len(initial))
        initial, _ , _  = sampler.run_mcmc(initial, nburnin, progress=True)
        sampler.reset()
    print("Running production...")
    pos, prob, state  = sampler.run_mcmc(initial, nrun, progress=True)

    return sampler

The recent one has more control because the scale in the random increments are defined for each one, but this one shows the case.

[crimson jackal]

The error function its just constraints and the "likelihood" computation.

[crimson jackal]

Sorry its getting to late for me, need to go sleep. I hope we can continue this later. If possible.

[wraith gust]

Hello how are you all

Anyone working on pandas and metapolib

[gritty vessel]

So gues I had blurry predictions before

So I tried few models like Unet2d Unet3d and Convlstm

In all I see the same pattern that loss stays nearly same

And model does not learn instead just gives blurry predictions

Can it be data?

[gritty vessel]

Sorry my mobile died

I'm passing 8past frames and trying forcast next 3 timestamps

Single channel and data is between 3 to 400kelvin so regression task

Max I have tried 25 epochs or so should I train it till like.50 or 100 epochs?

[gritty vessel]

This one currently I have kept for 30 epochs

Let's see how it goes

[polar current]

Hey I am trying to build a real time cctv based attendance system. Can anyone suggest me what face recognition models to use for my purpose?

[hollow cobalt]

It can be due to dataset prep, but sometimes responses can be accurate with non-optimal loss scores

[gritty vessel]

I did plot actual Vs predicted

[hollow cobalt]

Have you done qualititave anlysis on responses?

[gritty vessel]

No

Tell me more about it

It's numerical data

[hollow cobalt]

Well sometimes during training or subsequently, you will be able to score the responses using different techniques.

[gritty vessel]

Regression

[hollow cobalt]

Wait so your only raining for quantitative data eval?

[gritty vessel]

Yeah

[hollow cobalt]

How many model params does it have?

[gritty vessel]

Forcasting using historical data

[gritty vessel]

Standard one

Starts with 32 goes toll 512

And 1024 bottle neck

For unet 2d channels -8 giving past 8frames as channel and output is 3 channels that is next three frames

For unet2 single channel

Input depth =8 and output depth =3

Images are otlf size 128,128

[hollow cobalt]

32 Million?

Ohhh

Well it very interesting but I thought you were reffering to a more recent architecture not CNN's

[gritty vessel]

I tired various architecture tbh with different optizers and loss functions

But yeah it all resulted in blurry predictions

[gritty vessel]

I did try convlstm as well

[hollow cobalt]

Are you focusing on text-to-text generation or more multimodal focused?

[gritty vessel]

Txt to txt?

I am working with images

Transformers I have not tried yet

I will do that soon but what I have read from papers people have used unet2d unet 3d and convlstm to forecast and they got pretty good results as.well

[hollow cobalt]

I would suggest a small GPT model to run tests then as results will be more prominently effectuated

[hollow cobalt]

I mean it seems to be interesting I must say

[gritty vessel]

Gpt models are more text data right?

[north wolf]

just wanted to say, that i tried polars and i loved it.

[nimble mesa]

Hello
I need help with quantum computing. Does anyone know anything or can help me?

I want to do something like this but I get an error
I'm new to this and I want to learn little by little.
(I don't speak English well xd)

[nimble mesa]

@vocal cove

[vocal cove]

What error are you getting?

[nimble mesa]

qiskit_runtime_service._resolve_cloud_instances:WARNING:2025-09-10 06:06:31,689: Default instance not set. Searching all available instances.

QiskitBackendNotFoundError Traceback (most recent call last)
/tmp/ipython-input-3776664698.py in <cell line: 0>()
10
11 service = QiskitRuntimeService(channel="ibm_quantum_platform", token="mi token p :V")
---> 12 backend = service.backend('ibm_kyiv')
13 qc_basis = transpile(circ, backend)
14 job = backend.run(qc_basis, shots=4)

/usr/local/lib/python3.12/dist-packages/qiskit_ibm_runtime/qiskit_runtime_service.py in backend(self, name, instance, use_fractional_gates)
778 "https://quantum.cloud.ibm.com/docs/en/guides/qpu-information#view-your-resources"
779 )
--> 780 raise QiskitBackendNotFoundError("No backend matches the criteria." + cloud_msg_url)
781 return backends[0]
782

QiskitBackendNotFoundError: 'No backend matches the criteria. Learn more about available backends here https://quantum.cloud.ibm.com/docs/en/guides/qpu-information#view-your-resources'

[vocal cove]

Seems like the backend you chose is not available.

Print the list of all available backends and make sure.

[nimble mesa]

como se hace eso?

how do you do that?

[vocal cove]

Sign in first. You may not have access to some based on your region.

service = QiskitRuntimeService(channel="ibm_quantum_platform", token="mi token p :V")

Is that actually what you passed as the token?

[nimble mesa]

that part is already

[nimble mesa]

no

mi token is another

(I don't put it for security reasons (in discord))

[vocal cove]

Okay, just making sure.

[vocal cove]

Based on this, you don't have access to kyiv QPU.

[nimble mesa]

Change that one now, it still gives an error

[vocal cove]

Did you change to sth that is available to you?

Again, make sure you're signed in.

[vocal cove]

Okay, did you choose one of those?

[nimble mesa]

Is the only thing I need to change "service.backend"?
Or should I change the channel as well?

[nimble mesa]

yes

[vocal cove]

Just service.backend.

[nimble mesa]

qiskit_runtime_service._resolve_cloud_instances:WARNING:2025-09-10 06:18:02,122: Default instance not set. Searching all available instances.

QiskitBackendNotFoundError Traceback (most recent call last)
/tmp/ipython-input-2405784042.py in <cell line: 0>()
10
11 service = QiskitRuntimeService(channel="ibm_quantum_platform", token="private password :V")
---> 12 backend = service.backend('ibm_pittsburgh')
13 qc_basis = transpile(circ, backend)
14 job = backend.run(qc_basis, shots=4)

/usr/local/lib/python3.12/dist-packages/qiskit_ibm_runtime/qiskit_runtime_service.py in backend(self, name, instance, use_fractional_gates)
778 "https://quantum.cloud.ibm.com/docs/en/guides/qpu-information#view-your-resources"
779 )
--> 780 raise QiskitBackendNotFoundError("No backend matches the criteria." + cloud_msg_url)
781 return backends[0]
782

QiskitBackendNotFoundError: 'No backend matches the criteria. Learn more about available backends here https://quantum.cloud.ibm.com/docs/en/guides/qpu-information#view-your-resources'

[vocal cove]

Also, see if you can print the list of available backends via service.

[nimble mesa]

How do you do it?
Or with what command? Dx

Im in google colab
esta bien asi?

is it ok on google colab?

[vocal cove]

print(service.backends(min_num_qubits=64))

[vocal cove]

Then those two are it.

Try one of those.

[nimble mesa]

error again
qiskit_runtime_service._resolve_cloud_instances:WARNING:2025-09-10 06:23:11,750: Default instance not set. Searching all available instances.

IBMBackendError Traceback (most recent call last)
/tmp/ipython-input-524470724.py in <cell line: 0>()
12 backend = service.backend('ibm_torino')
13 qc_basis = transpile(circ, backend)
---> 14 job = backend.run(qc_basis, shots=4)
15
16 counts = job.result().to_dict()["results"][0]["data"]["counts"]

/usr/local/lib/python3.12/dist-packages/qiskit_ibm_runtime/ibm_backend.py in run(self, *args, **kwargs)
488
489 """
--> 490 raise IBMBackendError(
491 "Support for backend.run() has been removed. Please see our migration guide "
492 "https://quantum.cloud.ibm.com/docs/migration-guides/qiskit-runtime for instructions "

IBMBackendError: 'Support for backend.run() has been removed. Please see our migration guide https://quantum.cloud.ibm.com/docs/migration-guides/qiskit-runtime for instructions on how to migrate to the primitives interface.'

[vocal cove]

No, new error. That's progress. Okay, you see they say backend.run() is deprecated? Just open the documentation link to see what you need to replace it with.

Tip, qiskit uses primitives like Sampler, Estimator and such for this now. You have to use those.

[nimble mesa]

this

que no hay backend dice
that there is no backend

ok,now that's it

but here I have it like this

mm

[sick elm]

Hey folks, I’m working on a project that involves heavy math + Python (think linear algebra, probability, and algorithm optimization). Looking for someone who’s strong in both math and Python (NumPy / SciPy / SymPy / PyTorch etc.) to collaborate with me.

Remote, flexible terms — could be freelance or ongoing collab.
DM me if interested and share a bit about your background

[mild dirge]

!rule 9 6

[arctic wedge]

Rules

6. Do not post unapproved advertising.

9. Do not offer or ask for paid work of any kind.

[mild dirge]

@sick elm

[sick elm]

Here @mild dirge

[rich condor]

Has anyome ever worked with microservices systems on AWS? How did you build your data pipelines?

Am a complete beginner and need some tips for a project

[north wolf]

https://airflow.apache.org/

[modest vigil]

only if torch/gym could use polars dataframes, they are soooo fast

[agile cobalt]

well, it is not magic. You can use polars to load and transform the data, but typically the most performance intensive part when you're using torch will be in the GPU

There are to_torch methods in polars dataframes and series though, and there exists a GPU engine with limited support

[modest vigil]

are there ways to make torch really fast on CPU?

[agile cobalt]

do you know the difference in the amount of operations per second an average CPU can calculate vs that of which a GPU can?

at that level it isn't even really a question of software, but rather physical limitations

[modest vigil]

im tryint to stay CPU bound since my VPS is more cost effective than say runpod

[agile cobalt]

GPUs are orders of magnitude faster for some operations...

you wouldn't see companies investing billions of dollars into it if they weren't more cost efficient than CPUs, once you factor in their performance

(that does not applies to every single thing you can use torch for, but applies to training and running inference on a lot of models)

[vital basin]

https://tenor.com/view/mona-lisa-painting-gif-9441058

It's like 10000 cores vs 6

[modest vigil]

ok you guys convinced me im switching over to GPU, found thundercompute which is only $0.80/hr to run a A100

[nimble mesa]

I ended up changing the code to only use a simulation and not using my token.
Thanks for the help anyway, I was supposed to use "shots=4" to get 4 results (4 passwords created with bits and qubits)

According to what I read, there was an overlap, but I wasn't sure either xd

Many thanks to everyone who helped me
( @night dragon , @vocal cove , @final kiln )

!pip install qiskit qiskit-aer

from qiskit import QuantumRegister, ClassicalRegister, QuantumCircuit
from qiskit_aer import AerSimulator

n_qubits = 64

q = QuantumRegister(n_qubits)
c = ClassicalRegister(n_qubits)
circ = QuantumCircuit(q, c)

circ.h(q) #<-- puerta Hadamard

#hace que esten en superposicion, osea con valor de 0 y 1 a la vez
circ.measure(q, c) #aqui medimos la respuesta y se convierten en bits tradicionales

backend = AerSimulator(method="matrix_product_state") #El simulador p

job = backend.run(circ, shots=1) #Lo ejecuta 1 vez, o la cantidad que pongamos
result = job.result() #job.result obtiene los rresultados
counts = result.get_counts() #

random_binary = list(counts.keys())[0]
random_hex = hex(int(random_binary, 2)).upper()

print(f"Clave privada (binario): {random_binary}")
print(f"Clave privada (hex): {random_hex}")

[lusty iron]

I think you are misunderstanding what pytorch does. Polars uses multicore processing and query optimization for sql like data processing. Torch is for numerical computing and auto diffentiation. Torch will always be faster for things like dot products or array shape manipulations.

[steel spindle]

Hello, i have recently started making a simple chatbox. It's going well but I have a question about how can i make it so if I write "hwello" it would still understand that I'm trying to say hello? whithout creating a list of mispelled words?

[weak fog]

did you give it any system prompts?

[jaunty helm]

elaborate on chatbot
if it's any competent LLM then you don't need to do anything

[mild viper]

I'm looking to identify the optimal parameters, such as temperature and system prompt, for a pre-trained model solely through testing (without any fitting or training). I came across GridSearchCV, but it seems to be more suitable for tuning hyperparameters during training, with parameters specific to the training process rather than the model's behavior. Is GridSearchCV the appropriate tool for this, or is there an alternative? I'm also considering using nested for-loops to iterate through different parameter arrays, test the model, and calculate the score.

[waxen kindle]

The thing you are considering is exactly a GridSearch, and I don't see any other option than doing that

You can also add several checks to avoid testing some combinations of parameters, if for example a higher temperature usually leads to poorer results, then don't try very high temperatures

but I don't think there is much more you can do

[mild viper]

In that case, I suppose I'll make the "fit" method of my model essentially do nothing. Alternatively, is there a way to instruct GridSearch to skip calling the "fit" method on my model?

[waxen kindle]

idk what you are using, the GridSearchCV of scikit-learn ?

[steel spindle]

Whats LLM?

[steel spindle]

What do mean by system prompt?

[jaunty helm]

stuff like chatgpt, gemini, etc.
you should elaborate on what you mean by "chatbot" if it's not an llm, as that's the current trend and if you say chatbot without any other context people will assume you mean llm

[steel spindle]

It's not llm but i want it to understand and expore diff possibilty lika a llm

[jaunty helm]

so more traditional approaches without machine learning I assume
you're doing fuzzy matching basically
perhaps you can treat words within some small edit distance as the same

[steel spindle]

This is the code i have so far

https://replit.com/@-THE-CODER-/PythonFile#main.py

[mild viper]

Yes I’m using that one

[mild viper]

The 'cv' parameter of GridSearchCV must be an int in the range [2, inf), an object implementing 'split' and 'get_n_splits', an iterable or None. Got 1 instead.

However, since I'm not planning to perform any fitting, I don't require cross-validation. I intend to test on the entire dataset. That’s why I want to set cv=1

[waxen kindle]

That's bc you are no making a gridsearchCV

You actually only want a few for loops

[mild viper]

Yes I guess so. I’ve stumbled upon this “ParameterGrid” which I think is better than nested for loops. Do you agree?

[waxen kindle]

Never used so I can't say, but I wouldn't be surpised if it was nested for loops with extra printing and similar stuff

[prisma storm]

I'm interest. dm me

[serene scaffold]

The message you're replying to was against the rules.

[arctic blade]

Ive made a gradient descent program to calculate the line of best fit between 2 variables - it works, however as soon as the data within my dataset gets numerically large, the learning rate must decrease dramatically for my program to actually converge

Is there a way to know how small the learning rate should be?

Is normalising the data an option? Because this i tried, however it only worked when the numbers in the data were of similar magnitude

[tidal bough]

If I remember my numerical methods course correctly, yes. The learning rate must be lower than 2/λ where λ is the biggest eigenvalue of the hessian, something like that.

[tidal bough]

yeah, see e.g. here: https://chinmayhegde.github.io/introml-notes-sp2020/pages/lecture3_notes.html

[wide carbon]

Hello

Hello i need help for data generation for training an ml model so that diversity is ensured and clustering does not happen i have 6 parameters two discrete and 4 continuous which i need to vary

[arctic blade]

Thanks, and uh, what the hell is an eigenvalue and a hessian?

Also couldnt i just normalise it so the data is 0-1 and reverse the normalisation afterwards?

[waxen kindle]

you can

[waxen kindle]

some linear algebra concepts

[tidal bough]

A hessian is broadly speaking the matrix of second derivatives - for the linear fitting problem it's X^T X. What an eigenvalue is is, uhh, not that easy to explain, that's getting deeper into linear algebra.

[arctic blade]

Oh right, yeah thats waaayyyy beyond me rn

[tidal bough]

Also couldnt i just normalise it so the data is 0-1 and reverse the normalisation afterwards?
I think yes, though even after normalization the optimal learning rate will slightly depend on the data.

[arctic blade]

Right

Thanks

[waxen kindle]

a good practice is to schedule the learning rate to decrease after a few epochs

[wide carbon]

hello any help

[arctic blade]

Whats a epoch

Theres a lot of fancy words in this field huh

[waxen kindle]

a training step

an update of the model

basically your algorithm is:

learning_rate=...
for i in 1 ... N: (or with a while to check proximity)
update model(lr=learning_rate)

each loop iteration is called an epoch. N is the number of epoch

so instead you can have
for i in 1 ... N: (or with a while to check proximity)
update model(lr=learning_rate)
learning_rate/= 2

[arctic blade]

I could, but tbh this was less about optimisation and more just because if the numbers in the dataset were too big and the learning value wasnt small enough, id get ‘nan’ or ‘inf’ values and itd never converge

I figured out how to reverse the normalisation of the gradient tho

So it does work now, but thanks for all the fancy words anyway - i have a lot to learn huh 😅

[shadow perch]

Hello, I'd like to start creating AI, I wanted to start off easy and make an AI that can balance a pendulum but I don't know how or where to even start, I have the interactable GUI done with all the necessary physics and I added a score function. can anyone help?

or should I paste this into #1035199133436354600?

[waxen kindle]

so you want a reinforcement learning agent ?

[shadow perch]

yea

or an evolutionary network that learns

[waxen kindle]

I suggest to find a course on youtube

https://www.youtube.com/watch?v=ISk80iLhdfU

Thats one for example

[shadow perch]

I've tried but I can't learn from it, is there like a python ai from scratch kind of tutorial?

[shadow perch]

I'll try this

[waxen kindle]

idk, google it

looks like hugging face has one https://huggingface.co/learn/deep-rl-course/unit0/introduction

[shadow perch]

I'll do so, but I've seen like 4 hours of videos on reinforcement learning but I haven't caught anything from it

[waxen kindle]

did you take notes

are you actively watching ?

[shadow perch]

i tried to

[shadow perch]

I try to split it into sections

I understand the base core or the principle on how it works but I don't understand the way it shifts values and the error scoring function

[waxen kindle]

well, idk, if you can't learn with lectures, try written resources like this

[shadow perch]

I'll check it out, Thanks!

[waxen kindle]

I assume this also have some examples you can use as reference

[shadow perch]

yea, I think examples help a lot, I'll try to learn from the videos and this, I'll try to take a lot of notes like you said

I'll come back here soon

[waxen kindle]

sure! have fun

[shadow perch]

Thanks! I'll do so

[wooden sail]

for lipschitz continuous functions, yeah (since that 2/lambda value is the reciprocal of the lipschitz constant of the gradient of a linear least squares problem)

[abstract wasp]

hi i need help w a survey, i need like at least 50 responses (the ideal would be like 100 but weve only got like 25 :(( ), we cant get more ppl to take it, what would be some good strategies for collect more data. is it bad practice to ask them to take it again? could we synthesize the data? any suggestions?

[agile cobalt]

if you have no restrictions on who can answer it, use something like SurveyMonkey
if you have restrictions about it, consult whoever created those restrictions

asking the same person to ask again would be likely be pointless and lead to the same result as just duplicating their previous answers, unless "same person changed their answers over time" is part of the study

"we cant get more ppl to take it" is not a valid reason to disregard the data integrity, getting more people to take it is your only option to collect more real samples

synthesising/extrapolating/repeating rows could work if your goal is to use it to train a model, but is a no-go if you want for it to have statistical significance

and make sure to include where you sourced the survey answers from if you draw any conclusions based on the answers, even in part as a disclaimer of any possible selection bias

[abstract wasp]

Yeah we mainly want the data for model training
Well how do I get more ppl to take it 😭

Some one here said I can’t share surveys

[agile cobalt]

if you have no restrictions on who can answer it, use something like SurveyMonkey
if you have restrictions about it, consult whoever created those restrictions

and yes, sharing surveys here is against the rules. Same goes for most servers, that counts as spam/advertising.

[sudden whale]

i'm building an open source finance terminal similar to bloomberg terminal , it is under development but still useable - https://github.com/Fincept-Corporation/FinceptTerminal , please check the repo and make a star to support the work , any questions are welcomed

[distant spindle]

Hi, I'm new to leaning python and currently learning the basics from the CS50 course. However, as i want to specialise in the data science field, any recommendation as to what i should learn next and from where.

[wise ingot]

hi guys did anyone try out the mooc.fi course "Data Analysis with Python" and can tell a thing or two about it?

[weak fog]

its like giving your chatbot instructions

[ornate inlet]

Guys

[serene scaffold]

What? It's easier for everyone if you just say what you want to say.

[ornate inlet]

mb gng i forgot

[steel spindle]

Oh ok

[arctic wedge]

Click here to see this code in our pastebin.

[indigo bear]

I'm expecting to get some strategy close to 0.33 obviously when using CFR for RPS but I just get random numbers it feels
is 5 million trainings just not enough

[sterile cypress]

Hola

[jade blaze]

Hey does anyone know the package RDKIT? for computational chemistry

[mild viper]

Hey

[serene scaffold]

Always ask your actual question. Don't ask to ask.

[mild viper]

If I pass custom scorer functions to GridSearchCV, will it call the estimators’ predict method for each score? Wouldn’t it be more efficient to call predict once and pass its results to the scorers?

[lime grove]

don't you think that their own forum might be a better resource for something as specialized as this?

[final badge]

Is maths really important in ML?? if so, what maths is involved?

[serene scaffold]

ML is entirely applied math, and if you're not applying that math, you're not doing ML in the sense that companies are willing to pay top dollar for.

It's mostly probability/statistics, calculus (especially multivariate differential calculus), linear algebra, and graph theory

[final badge]

can u briefly explain how they are used in ML, if u dont have time then its fine .

[serene scaffold]

I'm at the gym right now. You can probably look it up.

[final badge]

alright

ty

[calm cipher]

a lot of machine learning models are elaborate curve fitting, understanding what that is and how it works is really important to understanding what you can and can't do with machine learning and what to expect from the results

all neural network operations are matrix multiplications, and understanding what they do requires you to understand the underlying matrix operations

classification tasks involve probability

applied machine learning research is essentially empirical research, so understanding statistical significance can help you understand how helpful different techniques are when presented in research, or for your own use when you're trying to show one model is better than another

it's useful to be able to read and understand mathematical formulas

I could go on but hopefully that's a good summary of the big things

[indigo bear]

does anyone know any good blogs or videos on understanding Rock paper scissor nash equilibrium with CFR?

I am struggling to understand how to implement it

[snow sage]

Large Language Model

[snow sage]

Yes ofcourse.. Maths is really important...

[steel spindle]

How is it?

[small wedge]

Not necessarily, not all ML is approximating functions. For example an RL algorithm that finds the best action to take and adds it to its sequence.

ML is any algorithm that can improve at a task without being explicitly modified by the dev

How does it not make sense?

Let me flesh out my RL example better and you tell me how you would characterize it as a function approximation then.

The initial training is the part where the algorithm improves without being modified by the dev no?

I never said it has to constantly improve, just get better at a task without the algorithm being changed by hand

Youre saying there's some ML algorithm that may not necessarily improve itself at all?

Okay so we have a platformer game, valid actions are left,right, and up denoted by characters L,R, and U. The model stores the sequence of actions it will take in a string. The policy simply increases reward as the model grows closer to the goal. The model continues to check all 3 new actions and append the one with the highest score until the level is complete.

How do we characterize this as function approximation

Hm I mean ig its a fair point, but if the model doesnt improve at all it has done no learning. I think the point of the algorithm is that despite there being no guarantee of improvement, stochastically we will eventually get some.

The improvement was the initial training process though? Maybe we're not on the same page here, a regular neural network applied to a supervised learning task would be an eager learner right? Because its done approximating the function before its ever used for inference?

Sure, but its still an ML algorithm that can be used for that task. Can it be characterized as function approximation or not?

Not talking about the chess example

I know that works for it

[small wedge]

Im talking about my example here

Okay, that is not excluded from my definition. The original stage of training is the self improvement part. Just because its not guaranteed to improve on one epoch/generation/full training cycle, it doesnt mean that the algorithm itself cannot be characterized as being able to improve by itself.

Im aware you can, I'm giving you an example that I dont believe is easily characterized as function approximation to prove my original point. Not because its an ideal algorithm for the task.

How so?

Its a policy I've defined, im not approximating anything

Its a fully defined function

Again, I am not talking about DQN

I am talking about my example, for the reasons I've given

It is iterative isnt it? Linreg models "learn"

They dont just spit out a prediction and you accept it do they?

Learning the sequence of actions to take to beat the level

Oh I see

You can run a linreg on a single data point, the "learning" would be iterating over multiple data points to approximate the best fit

Yeah i concede youre correct there.

Im not sure if you could even call the closed form machine learning then

Is it even learning at that point if there's no iteration? Is it not just directly calculating the result?

Are all forms of function approximation ML in your eyes then? And if not what distinguishes closed form linreg from me calling x + 2 an ML algorithm as it is technically approximating a function, if poorly.

Okay then let's use x + random.random() as the example

Thats an ML algorithm?

It has variability

Lol every example i think of becomes more and more obtuse

rolling = 0

out = dataset[i] + rolling

rolling += dataset[i]

This is a function approximation, that is not constant, and the approximation changes based on the input.

This is ML?

Im getting at the principle, none of this is supposed to be practical.

Alr fair enough, honestly just wanted to see what the minimum spec for an ML alg was in your eyes

Well

You call closed form linreg with no iteration training?

Wasn't your whole argument about it not being iteration that its not improving

Therefore not training

Right but I never specified it had to be iterative optimization, I just said it had to improve without being modified by the dev

I dont see how it can both be training in closed form but not fall under the definition of improving

Im mixed, I think i can switch the word improving for training and I have no problems with that. But just because an algorithm doesnt guarantee that its going to always improve, the reason we use it is because it can improve. Otherwise we'd never call it an ML algorithm in the first place. In other words the failures to optimize are the exception not the rule.

Ig either way my definition needs to be more concise.

Either training or has the capacity to improve

Ofc, but i dont think "generalizes well" is in the definition of ML either, they can be overfit all to hell and still be ML.

Or even in improves

We've moved again from principle to practice though

In principle an algorithm that can improve on its training data is ML regardless of its ability to generalize

Given all the other shit yk

Dont wanna write the wall of text every time that was my definition of ml

I know

I was talking about definitions. None of what I said applies to practice.

But to wrap it up, I think its fair to characterize all ML as function optimization. But I do still think my definition of ML is fine (maybe given a few tweaks to the improvement part to make it clear that it doesnt have to always lower the cost)

[serene scaffold]

@bronze wyvern a lot of things you just have to develop a sense for. is it more important to know how much something deviates from the mean, or where it sits within the range of possible values?

[bronze wyvern]

yep I see, I wanted to understand these 2 a bit more, like how the statistics work, why one is better than the other in a particular scenario or the other is better in another scenario, is there something I can read for that pls. I have a knowledge gap when it comes to these 😭

[serene scaffold]

there are resources in the pins. I don't have one in mind for that particular question.

[bronze wyvern]

noted, will have a look, ty !

[bronze wyvern]

The thing that confuses me is I want to know, what is special with the mean or sd that it allows us to standardise our values and what is special when we center something based on its mean

[bronze wyvern]

hmm the mean is the expected value we want? If we don't get the mean, if we consider z score, it would mean that our value is greater or less than 0. What does this implies if we have a negative or positive value?

For the sd I didn't understand, we divide by sd, what is the consequence/effect of that?

yep I see

Dividing by the standard deviation tells us how many standard deviations the numerator is (the difference in our observation and the mean).
Why does it matter? what can we conclude out of that?

hmm still a bit confusing for the sd :c, I'm confuse of how we would interpret it because the z score depends on both sd and mean not only one of them

I know z score = value - mean divided by sd I think

it's greater than the mean so our observation are greater than the mean

[cursive totem]

hey guys, i wonder how can i get accuracy score for data, where each prediction looks like this? there is like a lot of points for each prediction, so i dont even know how to use any different metrcics but loss

sometimes prediction looks like this so i dont even know how to tell the code which prediction is accurate and which is not

and btw, is there any way to not use system disk c? i even did this at the start of my code, but it just destroys my disk c anyway...

os.environ["TORCH_HOME"] = r"D:\cache\torch_cache"
os.environ["TMP"] = r"D:\cache\tmp"
os.environ["TEMP"] = r"D:\cache\tmp"

even powershell doesnt see where my memory disappearing

[calm cipher]

which metric are you currently using?

"loss" just refers to the loss function, which could be any number of different metrics

to put it another way, it looks like you're trying to solve a regression task where a strict notion of accuracy doesn't apply because it's applicable in classification tasks

[runic scaffold]

how to do constraint based sampling ?

[cursive totem]

Mseloss currently

[jaunty helm]

what's wrong about mse though?
usually accuracy is used to evaluate classification rather than regression like you have here (and it's not even a great measure)

[cursive totem]

Yeah i know i just wondered is there any other way to estimate model

[jaunty helm]

mse works fine already
you should also look into the specific field you're working with and see how others measure performance
e.g. maybe your metric can be the distance function (that may be specific to your field) between the true spectrum and predicted

[wooden sail]

maximum (log-)likelihood is pretty common. this includes the MSE if you know that the data has a deterministic mean and suffers from AWGN

[cursive totem]

The thing is it is my scientific work for diploma and there is not much information about that, so i am just experimenting with it and trying different ways to evaluate

[agile cobalt]

idk about Raman shift in particular, but there is a bunch of research regarding Spectrograms overall?
from text to speech to time series forecasting

[spring field]

that's certainly an interesting approach
and the mentioned plot image approach as well

[leaden hull]

As the question is not really python-specific I, as a chemist I felt free to react. If this doesn't belong here, I am sorry, I am just herer trying to help.

If it is for a diploma, I assume you have a professor (or similar) that you work under. You could ask them for some starting literature regarding this specific topic. That would always be my step one.

I have done various projects that involve scoring accuracy for prediction models, for example segmenting images using an AI tool and comparing that to human segmentation. What I have learned is that the specific metric that is used doesn't matter as much and tends to be subject-specific. Every field of study tends to use their own preferred systems. So it is more about what the metric represents, which differs case by case. So in my example, I was interested in knowing how my AI model differed from a human model, assuming the human model was 100% accurate. For this i could use simple confusion matrix scores.

Now for your system, you are interested in how the prediction compares to a known quantity at a given wavelength. However, it is unclear to me what specifically you want to measure. For example, if you are only interested in performance relative to the real values, you could take the difference between each predicted and real value and use that as a measure of accuracy (e.g. RMS, R2). However, suppose your predicted spectrum has a 50 cm-1 shift, then the score of this metric will be worse than you would expect, so you would need to define it differently.

Now since ramann spectra tend to only care about the peak values, you may want to use a metric that only compares the peaks, as the rest is considered noise. In this case, you could compare the peaks of the prediction with the peaks of the model and finding the average shift between the predicted and real peaks.

So all in all, I think the best way forward is to first determine for yourself what it is exactly that you're trying to measure.

Because calculating a 'loss score'' gives you different information about the performance of your model than using something like a peak position error score, which in turn gives differnt information from using a confusion matrix type scoring

TLDR: There is not one metric that is the best in all use cases. Different types of scoring gives you different types of information. Try to determine first what type of information you want. That will help you in determining the best way to evaluate the accuracy of your prediction.

[digital valley]

hello

[cursive totem]

Thanks, that was valuable advice

My supervisor told me to just experiment with different loss functions and told me that i have 2 weeks to get some result

Tbh i study at applied physics and know less about raman spectra than neural networks that i learned in few months lol, maybe that's the case here :D

[snow sage]

You got so many Explanations from here...I Guess... Am i right?

[snow sage]

Hello>

[steel spindle]

Yes

[night dagger]

Hello everyone,
I am working on a taxi mobility platform and exploring AI features such as demand heatmaps, driver acceptance prediction, and dynamic incentives using open-source tools.

May I ask for your advice on the best way to model and update these in production? I would also greatly appreciate any suggestions for additional creative AI features that could enhance such a platform.

[calm cipher]

To add a few more details to this is that mean squared error isn't bad for training loss because it penalizes large differences more than small differences, but it isn't as intuitive when trying to evaluate how off your answers are

you might consider root mean squared error (RMSE) which makes it more like distance, which is much easier to think about

but having the domain knowledge that the peaks are the most important part of the output is good to know, if that is the case then it makes sense to focus on them

I would also say that if non-peak areas are noise then it might not make sense to model them, if the model is learning to output noise then I wonder if it's actually picking up on any patterns in the data

[snow sage]

Hehe

[void cape]

Hello, I have this project on virtual stock trading project (we can use any strategy). ig the goal is just to get them gains
I see this as an opportunity to apply data science to analyze and create decisions based on those.
I planned to use ML models but have read that it can't be at all reliable.
That's why I only opt to analyze data by visualizing and correlating and stuff like that.

‎
Do you think this approach is appropriate for my project?

[Gather/clean historical data > compute return/risk metrics > decide which stocks are good to focus on > plot em > find correlations (for diversification) > find spots/signs to base buying/selling > simulate/test > allocate investments > idk for now maybe some conclusion stuffs]

‎
left out many details but thats the gist of it

[leaden hull]

The raman spectrogram essentially is just a probability distribution of photon wavelengths that correspond to various vibrational modes of a molecule. So a peak in the graph means that more photons of that wavelength were detected by the detector. Now the difficult thing about the noise here is that if we measure a vibrational mode that's more rare, than the peak in the graph corresponding to the wavelength of that bond will be lower. So if you for example use an AI, it can be hard to differentiate between small peaks or noise.

Thats why I said you tend to only use the peaks, but in reality it's always more nuanced.

[calm cipher]

ah I see, that's interesting. I guess the problem with modeling in this case is that if the data you are learning to model is noise then there is no pattern for the model to output

it may be better to treat it as an outlier detection problem rather than a sequence modeling problem, or to model it in a way where the non-peaks don't contribute to gradient calculations

I was wondering if modeling it as a sequence of binary peak/non-peak outputs would make more sense

[leaden hull]

That would depend on what information you would want out of it really.

If you are interested not only in whether you find a peak, but also about the relative size of each peak, then you wouldn't be able to do a binary output.

[calm cipher]

my area is more speech and dialogue but there's a model architecture from TTS that might be relevant here

you jointly predict output speech signals and a binary gate value predicting whether the model is finished outputting speech

[leaden hull]

But if you only care about correctly identifying the wavelengths than yes that would totaly work!

[calm cipher]

and then you only compute the gradient and do backpropagation from the speech output frames where the gate is supposed to be true

then for purposes of backpropagation the model ignores any outputs where the gate is supposed to be false

That might be too complicated for this honestly but something like that might allow the model to ignore the noise when learning

[leaden hull]

I'm not going to lie, I have no idea about how AI works. I just wanted to help them understand this in a broader sense.

[calm cipher]

ha i see, it's interesting hearing about the problem

but it does sound like it might be worth trying to avoid modeling the noise parts

[leaden hull]

It depends on what their objective is with the model

You could even use a two-metric system to score the graph. For example, you could use one to score intensity and one to score wavelength

[grand minnow]

How do you host your MCPs? As Docker containers or as SSE?

[agile cobalt]

how would you host in Docker without using SSE?

I'd argue for either using stdio transport with uv run, uvx (or the npm equivalent if you're using third-party servers made with JavaScript) or hosting a server for SSE transport the same way you'd host a normal API, whenever you use a container for the mcp server or host it on the same machine/container as the agent could depend on the server though

(if the server is very tightly coupled to the agent, shares dependencies and is only ever consumed by that one agent, might as well put it together. If multiple agents talk with the server, host it separately)

oh wait, later versions replaced SSE by Streamable HTTP?
same idea though

[grand minnow]

Would it make sense to host all of the MCPs as a separate server and be connected via SSE? Then the agents I have just connect with whatever is available?

Not aware of the Streamable HTTP

[agile cobalt]

doesn't sounds like it is a special thing, just make SSE optional (optionally using normal requests instead) https://modelcontextprotocol.io/specification/2025-06-18/basic/transports

[grand minnow]

I see

[agile cobalt]

what exactly do you mean by all?
I would keep some degree of isolation such that you can decide which toolset makes sense for which agents, you don't want to throw in dozens of irrelevant tools

[grand minnow]

💯

What I meant was that I have MCP servers of various kinds, running on uvx and npx and docker.

I was thinking of setting them up in one server as either docker containers or as SSE (or maybe I could switch over to Streamable HTTP)

And then the agents just fetches from that MCP server specifically as instructed, but not all per agent

so it would be like bla.com/github and bla.com/jira or something and then the github agent would connect to bla.com/github, and Jira agent to Jira

[agile cobalt]

I mean, the alternative to SSE/Streamable HTTP is stdio, and personally I would rather avoid using stdio over docker

if you want to dockerize it go the HTTP route

[grand minnow]

Right. I guess I'll give that a go

[long meteor]

Can anyone recommend books or smth to learn about rl and unsupervised learning more in detail? Especially about implementing it in pytorch or Scikit-Learn

[agile cobalt]

have you gone through scikit learn's user guide yet?

they also have a course but it iirc only talks much about supervised learning

RL is a different beast though, maybe start at https://gymnasium.farama.org/introduction/basic_usage/

[spice tartan]

Yo

[oblique isle]

guys

hiii where can i find a good data for a Segmentation use CAse ?

[nocturne whale]

Hey Guys, I created a very simple AI Agent simulating Master Roshi from Dragon Balls that has access to the dragonball-api.com API. Agent was built using nomos - NoCode (Full code is also available, if needed more control) https://github.com/dowhiledev/nomos

Demo Link: Master Roshi - https://roshi-ai-showcase.vercel.app/ (Backend uses the Mistral Devstral small, which is a very small model, and render free tier to host - so expect slow responding) Frontend is full AI generated and uses the Nomos TS SDK

[supple shoal]

looking for someone who’s great at ml and has experience with aerial/drone imagery. dm if this is u or if u know someone

[glossy pasture]

hi guys

Someone is selling me chatgpt pro account for just 50 instead of 200 USD. He tells me codex cli unlimited with gpt-5-codex is the best to code python. Should I get it?
Is it good offer?
He gives me free trial

[long locust]

Sounds like a scam man

[glossy pasture]

I thought too but it really works he gave me free trial it works

Do you think it is worth it?

[long locust]

No, I think it is just a straight up scam

[glossy pasture]

It is not a scam i can use it and it works

[long locust]

If something that costs $200 is being given to you for $50, you are being scammed

[stoic gorge]

And how can you be sure that trial isn't being sold to multiple more people? You'll eventually all end up with the same chatgpt account and get banned

Give nice thing for free to bait into buying fake thing is like the most basic techniques used by scammers

Bonus points if that paid account is bought using a stolen card. Also really common in scams - will work for some time, then you'll get banned

[glossy pasture]

Idk he gives me one complete month trial it doesn't look like a scam and I can't see other conversations besides mine

He got discounted account in other countries

Also sells plus por 23 usd for 6 months

[long locust]

"If something is too good to be true, it probably is"

Just use the free versions and don't get scammed out of your money man

[glossy pasture]

I can give you the trial he gave me if you want to check

[long locust]

No

[glossy pasture]

ok thanks

[agile cobalt]

odds are they are either using a stolen credit card, or the free trial is bait to scam you
probably both tbh

[real smelt]

Hi I’m starting my bachelors in AI and DS , any info that I get

[vestal lantern]

hi guys anyone there

i want someone whos know how ai code checker workers

[serene scaffold]

What do you want to know about them? It's better to always ask your actual question

[mortal shoal]

Hello! I have a Speech-To-Text Project: Docker/Docker-Compose/Frontend with Svelte/Backend with Python.
I use the MediaRecorder API in frontend (browser).
The recorded audio file (audio/wav) is passed to the backend as follows:

...
const file = new File([blob], 'chunk.wav', { type: 'audio/wav' });
const formData = new FormData();
formData.append('audio_file', file);

const response = await fetch(`${this.backendUrl}/transcribe`, { method: 'POST', body: formData });

....

My question: What is the best approach to convert in the backend UploadFile --> np.ndarray?

[serene scaffold]

https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html

[mortal shoal]

Thank you. Will try right now.

[simple flint]

hey any buddy can give me idea i am stuck with data inputting need to input data in json multiple table convert put lastly in sql database

[opaque condor]

Is there a dataset for a human detection in an image?

I've been searching

[obsidian talon]

Hm still need help?

[stable wind]

Guys iam tryna build an educational website but iam broke and i need an free api thag has like decent ing recognition to people can upload homework and stuff and I was planning to start on payrols after getting a handful of users anyone aware of such services thar have img recognition good enough for hw and doubt solving but has a small free tier?

[long meteor]

I created a project for comment classification, would be glad if you could review my code https://github.com/Akomon333/Comments-classifier-V2

[past meteor]

Two options:

1. stdio, that's what I'd use if the MCP server is running locally and needs access to my filesystem
2. HTTP. Go for streamable HTTP if you're using a client that supports it, SSE otherwise.

I never have had to make an MCP server that needed access to my filesystem so I always go with http.

Both can be ran as docker containers

[merry oak]

!rule 9 | We're not a job board. You're message has been removed.

[arctic wedge]

Rules

9. Do not offer or ask for paid work of any kind.

[fallow coyote]

Want to ask for any non software engineers how do you utilise python data analysis or ML techniques in your work? Im currently in an engineering apprenticeship which does deal in a lot of analyst based work. Even though, I dont really need to, the company actually are encouraging me to develop my programming skills

[lucid elbow]

Any one here with an accounting background who pursue data science later

[worldly dawn]

These are all means to an end.
Some ways to help you look at it would be to think about:

• Given the raw data available, would there be any combination or processing that would make the product smarter? For instance, imagine you are a the owner a discord server where users have tags for topics or languages. You could match people with similar interests, or recommend cool people, or detect key influencers, or detect spammers, etc.
• Given all that data available, what insights can you build for the benefit of the company? Reusing that example of discord server, you could try to understand better what make people leave or stay. Are people leaving after they got their questions answered? Do they then stay into topical channels? What's their journey?

And you could approach that from two ends:

• Ask around what other employees wish to know
• Given the data, what can you do with it

[fallow coyote]

I was also wondering from more of a mechanical/manufacturing engineering viewpoint thats the sort work Im doing (or more accurately, being trained up to do)?

[worldly dawn]

same ideas

[fallow coyote]

From what Ive been learning on the job, theres so many things I could code for. What Im going to focus on is brushing on my data vis skills (as my workplace uses excel for data analysis) and then data analytics to help me make better informed decisions with any work projects ill be working on

[worldly dawn]

Definitely!

Most often it ends up being a question of priority, reach, scope, impact, cost and availability of that data (or cost to acquire it)

[errant lake]

On my side, it's about finding anomalies in huge amounts of billing data. In terms of tooling, jupyter notebooks, seaborn, bigquery.

[fallow coyote]

Next week, Im going over to another city to shadow one of my coworkers doing some testing on a machine for a company. Ill ask him if he doesnt mind giving me some datasets related to it to help put my dsa skills into an actual use case

[past meteor]

My company has 100ish people doing data stuff, most of them using Python to some extent. I think recursive error names a lot of cases we do

[worldly dawn]

Unless they are in DE/DS/ML, it's unlikely for them to know about that and have it ready for you. It would also not be their responsibility.
I would suggest instead to probe them at a higher level:

• What data are they collecting?
• What data is available to them but not collecting?
• What has been difficult for them to accomplish?

Note also most folks focus on the specific of their jobs, for instance testing a machine. But there is a wealth of insight and data to derive from the "meta" level, about how they accomplish their job. For instance, is it always the same machine having issues? What is the MTBF? How far do they have to travel? Can we predict failures? In short, any insight that would help making the job more efficient

and if you can predict failures, then maybe you can ensure you have enough stock of parts for that machine by the time it fails?

[fallow coyote]

I'm definitely going to ask these questions next week. Never really thought about it that way

[fallow coyote]

Hopefully they do 😂. They use machines similar to where I work at and, they cost £350000 a pop!

[worldly dawn]

and if you know how long it takes between ordering a specific part and receiving them, you can alert on the danger zone since you know the machine is likely to fail within a timeframe that does not allow to ensure having spare parts and thus forcing the machine to be out of commission for longer than necessary

[worldly dawn]

you would be surprised

[past meteor]

These usecases are the hardest

And ones that I'd try to not do

[errant lake]

@fallow coyote Do you know already what kind of data warehouse the company is using? Where you will have to store and query that data?

[worldly dawn]

you would need enough machines and failures to have enough data for that too

[past meteor]

Yup

• it's a causal inference problem

Data science often fails because people bite off more they can chew

[worldly dawn]

It depends on the context too

[fallow coyote]

Not really. Tbh, I'm not really in that department. Im in their manufacturing development team. Essentially, we get the engineering designs and produces the models for them to be machined out. Theres a lot more to it as we also develop and design the best possible tool paths for models. I only want to use data analytics as I think itll help a lot in my role. It is a lot more data driven than you think it is

[fallow coyote]

Ill see how it goes. The simplest thing I can do is data visualisation. Get the data from excel, use python to produce visuals in the way I want to show them. Then, see how I can implement data analytics using python.

[errant lake]

Oh ok then it's really small scale. Look into jupyter notebooks, openpyxl, pandas

[past meteor]

Make sure your analysis is actionable, that's ime the thing people struggle with the most

E.g., don't show people data randomly, it needs to be stuff they can do something with

[fallow coyote]

My company is weird as despite it being a small company, we work with major companies in aerospace. The products we produce are only available to those companies as theyre very bespoke

[errant lake]

I meant small scale in terms of amount of data. One computer seems to be enough to handle the amount you describe

[next panther]

wtf is a grown ass man having an anime pfp💔

or at least a girl one

{not sexist btw}

[past meteor]

It's hard to give examples, but that's the thing my juniors struggle with the most - they tend to give me a lot of numbers if I tell them to do an analysis but it's typically not something I can do anything with

"You analysed X about feature Y, but you gave me no way we can improve the feature based on your analysis"

[errant lake]

isn't it never good to know you can't do anything about it?

[past meteor]

Let me give you an example, imagine your dataset is about flights at an airport, it's kind of irrelevant you just give me visuals and info about all the flights

Rather than, "Oh, I found flight X has long layovers, maybe we should put it closer to the food court" or whatever idk

[fallow coyote]

Ah that makes sense. The more experienced I get, Ill get onto doing more advanced stuff (as long as the company is fine with me developing something for them)

[errant lake]

I get that, but if you asked to improve the rate of planes landing and you found out there is no solution that can be implemented without putting people at risk

it's not actionable, but you know you're fully optimized on this point

[next panther]

am gonna make a cool python project

trust

[past meteor]

Indeed, that's fine as well

so long as you do your analysis with a goal in mind, if your goal is to improve something and you can prove it's as good as it can be that's an interesting outcome

[errant lake]

kinda edge-case, but never be ashamed of not having a solution, sometimes there is just none

[next panther]

bruh

[errant lake]

forcing yourself to give one leads to bad decisions

[past meteor]

No, I agree with this 100 %

[next panther]

why does it feel like am talking to myself