Legal Rag system

[bright iris]

I'm attempting to create a legal rag system using PHI-4 to answer any legal questions regarding UAE law and solve any user uploaded cases based on the laws. However when i attempt to ask a complex question the model answers mostly correct but always misinterrupts some of the answer, i also tried using nemo-minstral but with similar results. Any idea if this is a coding or prompt related issue? And if its a prompt related issue id appreciate if someone gives me a quick example of a good prompt i can use.

[vital moss]

First place to look at is to see if the articles being retrieved are actually relevant to your query. If not, the problem is with your embedding model.

[bright pine]

What's your parser or the initial dataset look like? unstructured data or structured into the RAG?

[bright iris]

What if some articles are relevant but some are missing and not being stated

[vital moss]

So you’re saying you are not seeing the exact list you want in your top-5 search/retrieved results?

Next question would be to ask why those articles are missing over the others. Are they an indirect extension of the prompt (ie you ask about circumstances around murder but info about manslaughter is missing)?

[bright iris]

Basically if i ask a complex question that requires answers from multiple articles it sometimes retrieves correct articles but with some missing and every now and then can mix up the article content

As for the questions i made sure they are generated correctly according to the context of the provided documents

[silver dust]

What kind of database are you using? If you're using a vector database, you might want to experiment with a graph database

[bright iris]

yeh im currently using a vector but was thinking about trying a graph instead

[silver dust]

yeah man you might want to try graph rag instead. ill give it a shot on my end with some docs and share what i see

[vital moss]

let us know how it works out. I have yet to implement a local graph RAG model on my own, so I'm curious to know if there are any good models for that.

[tired mirage]

Here’s a heuristic. If you use Sota models and it works, it’s a model problem. If you use sota models and it doesn’t work, it’s a program/orchestration problem.

[bright iris]

hello i know its been a while but i have been travelling so i didnt really get a chance to work on the project until now. I used the graph rag with neo4j and qwen as the response model and so far the answers have been way more accurate ranging in the high 80% for very complex cases and in the 90% for simple questions

[solar spindle]

Greetings! I'd be curious to discuss methods; if you're willing. Though I suspect you're much more knowledgeable on the subject than I am. That said, I'm using hybrid graphrag for my use case of an internal legal rag; with both many-to-one and many-to-many relationships along the graphrag. Or perhaps valuable legal related api?

[bright iris]

what would you like to discuss or know about?

[solar spindle]

What's your deployment method? Cloud or local? Are you using strictly graph rag or are you using a hybrid vector rag approach? And if you'll forgive my being nosy, where you're getting your training data from.