Is it possible to score a full dataset run? | Langfuse | Page 1

vagrant oxide Aug 13, 2024, 6:17 PM

#

I'm wondering if there is a best practice for evaluations that require multiple traces. Primary use case would be running a prompt against a full dataset and wanting to evaluate the total precision/recall/f1/etc. Right now I can score each dataset item but I haven't figured out a great way to surface metrics to the UI that would encompass the full run.

The alternative I've tested is encompassing the full run in a trace and scoring that but it seems a bit hacky.

wraith pondBOT Aug 13, 2024, 6:17 PM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

zenith vigil Aug 13, 2024, 7:34 PM

#

Can you add your +1 to this idea post that tracks this feature?

https://github.com/orgs/langfuse/discussions/2511

Currently only averages are nicely supported but we plan to look into run level scores

GitHub

Scoring dataset runs, e.g. precision, recall, f-value · langfuse · ...

Describe the feature or potential improvement In LangSmith, for example, there is a feature to get precision, recall, f-value in experiment feature on a dataset. https://docs.smith.langchain.com/ho...

vagrant oxide Aug 13, 2024, 7:44 PM

#

Done! Thanks for the quick response

zenith vigil Aug 13, 2024, 8:44 PM

#

thank you! sure, happy to help

#Is it possible to score a full dataset run?