#🔒 Matching Confusion
rotund sonnet
!paste
scenic fableBOT
rotund sonnet
use our pastebin
brittle nova
!paste
rotund sonnet
go the url, paste your code in there, press ctrl+s and share the new url here
brittle nova
k
The script was made to process a text file with data about events, tokens, and sentences. The goal is to:
Read the data and extract the event, token, and sentence.
Find the document in my database (Book2) that contains the sentence .
Locate the token within the document and get its exact position (start and end)( not managing to do it because it confuses it with other words).
Save everything — the event, token, sentence, document info, and token positions — into my database (Book2Info).
What I’ve Tried
Here’s my current approach:
Search for the Token Directly:
After finding the document with the sentence, I search for the token in the document using str.find(). To make sure it matches, I normalize the text and token by converting them to lowercase, removing accents, and trimming spaces.
Use Fuzzy Matching as Backup:
If the direct search doesn’t work, I fall back to fuzzy matching (with rapidfuzz) to try and locate the token.
The script finds the sentence in the document, the token — which is part of that sentence — should also be there. So, once the sentence is matched, finding the token should be straightforward.
Even though it sounds simple, it sometimes doesn’t work. Here’s what’s going wrong:
Token Not Found:
Sometimes the script says the token isn’t in the document, even though it clearly is. This happens even when the sentence is matched perfectly.
Issues with Normalization:
When I normalize the text and token (to make them easier to match), it sometimes changes things too much. For example, accents are removed, or spaces are altered, which makes the search fail.
Direct Search Problems:
Since the script searches the whole document, it sometimes grabs the wrong token if it appears earlier in the text than in the matched sentence. And if the normalized token doesn’t exactly match the text, it fails altogether.
rotund sonnet
Token Not Found:
Sometimes the script says the token isn’t in the document, even though it clearly is. This happens even when the sentence is matched perfectly.
Do you have a specific case of this happening you can share? It'll make it easier to pin point what's going wrong
brittle nova
[10] Processing sentence: "And in the same way, conduct an investigation into the said false weights, determining how long they have been used, in what manner, and what faults were committed against my estate and parties, and who is responsible."
✔ Sentence found with a score of 100.0
✘ Token 'knowing' not found in the full text.
[11] Processing sentence: "And because it is necessary to particularly know how and in what form the funds sent from this kingdom for the purchase of pepper were spent, what was spent on it, on interests and exchanges, on ship repairs, and what was allocated to any other purposes, and the efforts made to recover anything lost through any means, I strongly charge you to ensure that a very clear and detailed account is made from the year 604 to this point, which you will send me via the same channels you use to write to me, and ensure that each year a similar account is sent to the India Council, as well as the one sent to the India House."
✔ Sentence found with a score of 99.35838680109991
✘ Token 'know' not found in the full text.
This error occured when i tried to solve in another way. But involved the following part changed:
def locate_token_in_text(self, text, sentence, token):
normalized_text = normalize(text)
normalized_sentence = normalize(sentence)
normalized_token = normalize(token)
sentence_start = normalized_text.find(normalized_sentence)
if sentence_start == -1:
return None, None
sentence_end = sentence_start + len(normalized_sentence)
sentence_fragment = normalized_text[sentence_start:sentence_end]
token_start_in_sentence = sentence_fragment.find(normalized_token)
if token_start_in_sentence == -1:
return None, None
token_start = sentence_start + token_start_in_sentence
token_end = token_start + len(normalized_token)
return token_start, token_end
With the direct approach it links directly but if there is a match before it will prioritize that one. Because of that i tried to change the way the token is located starting from the specific start position where the sentence is, but it didn't find it.
rotund sonnet
[10] Processing sentence: "And in the same way, conduct an investigation into th...
Okay, so for 10, where is "knowing" supposed to be found there
For 11, the implementation you have in the pastebin does work for me.
For your approach that you posted here, what's the difference between text and sentence? Given what you have for 11, what would be the sentence?
brittle nova
Okay, so for `10`, where is "knowing" supposed to be found there
For `11`, the i...
I’m working directly with a database that contains historical documents. What I shared earlier was just a snippet; the full dataset has over 6,000 sentences. Each row in the database represents a full historical document, with the entire text stored in it.
For my task, I’m using individual sentences—small excerpts taken from the full text—to locate their corresponding context within the full document. These sentences are extracted from a separate .txt file called Book2Events.txt. Once I find the sentence in the full text, I use it as a reference point to link and locate the specific token associated with that context.
In summary:
The database contains full historical documents.
The .txt file contains sentences (smaller parts of the full text) and tokens.
The challenge is to match the sentences to their respective full texts and precisely locate the token within them.
rotund sonnet
I’m working directly with a database that contains historical documents. What I ...
So, for #10 there, was "knowing" supposed to be found there? Just trying to make sense of what the functions are going to be getting and the specific problem there might be
brittle nova
So, for #10 there, was "knowing" supposed to be found there? Just trying to make...
Yes, the token "knowing" was supposed to be found, but it wasn’t. I have a data structure that stores information about each event, including the token, its start position, and end position. The script is designed to fill this information automatically if it finds the matches in the dataset
Thanks for the help
rotund sonnet
Yes, the token "knowing" was supposed to be found, but it wasn’t. I have a data ...
Okay, so for #10 that seems like an issue with the wrong sentence being found or maybe even the intake of the events? Being the word "knowing" or even "know" is not in the sentence you have there for #10.
For #11, what is the issue with you original implementation in the pastebin? Because it does find the token for me.
brittle nova
I changed it from the original language, can i pm you with the true text?
rotund sonnet
Ah, if you want you can post the original language here if it's obscene or anything
brittle nova
[10] Processando sentença: E da mesma maneira façaes tirar devassa sobre os ditos pesos falsos, sabendo-se o tempo que ha que com elles se pesa e em que forma, e que culpa se commetteu n'isso contra minha fazenda e partes, e quem são os culpados.
✔ Sentença encontrada no documento: Documento 160. 1611 - Janeiro 21 com score 100.0
✘ Token 'sabendo' não encontrado no texto completo do documento.
[11] Processando sentença: E porque convem saber mui particularmente como e em que forma se despenderam os cabedaes, que se enviaram d'este reino para a compra da pimenta, e o que d'elles se despendeu n'ella e em interesses e cambios, concertos das naus, e se applicou a quaesquer outras cousas, e as diligencias que se fizeram sobre o que d'elle se tirou por qualquer via, vos hei por mui encarregado ordenardes que se faça d'isso hũa relação authentica mui clara e distincta, desde o anno de 604 a esta parte, a qual me enviareis com as mesmas vias que me escreverdes, e ordenareis que dos que forem cada anno se envie ao conselho da India a mes ma relação, alem da que se envia á casa da India.
✔ Sentença encontrada no documento: Documento 160. 1611 - Janeiro 21 com score 99.35838680109991
✘ Token 'saber' não encontrado no texto completo do documento.
Might everything go here it is the full sentences
rotund sonnet
okay cool, give me a few minutes to take a look
Curious. I am getting a match with the implementation you have in the pastebin
brittle nova
Well, that's weird its not finding in mine. What a bummer
rotund sonnet
If you extract out the noramlize and the locate_token_in_text functions into a separate script and run it with manually passing in the 2 examples above, do you get a match?
brittle nova
Yup, when isolated it worked
rotund sonnet
Okay, can you add a print or log statement, to see exactly what is being passed into the locate_token_in_text function? Like just add a print or log for token and text at the top of that function
brittle nova
Ok
[1382] Attempting to locate: 'Viso-rey da India amigo, eu el-rey vos envio muito saudar.'
✔ Sentence found in document: Documento 189. 1611 - Fevereiro 25 with score 100.0
DEBUG: Full text: Viso-rey amigo, eu el-rey vos envio muito saudar. Vendo a grande
falta que houve na pimenta que se e...
DEBUG: Sentence: Viso-rey da India amigo, eu el-rey vos envio muito saudar.
DEBUG: Token: saudar
✘ Token 'saudar' not found in the document.
something like this?
rotund sonnet
And is this with the function in the pastebin or the one you posted here? They're a bit different
brittle nova
The one i posted here. Where the locate_token_in_text changes and in match_sentences it is added: "token_start, token_end = self.locate_token_in_text(matched_document.content, sentence, token)"
rotund sonnet
[1382] Attempting to locate: 'Viso-rey da India amigo, eu el-rey vos envio muito...
Do you see the difference in the debug statements between the full text and the individual sentence?
brittle nova
Does one version have more characters than the other? This specific sentence is repeated throughout the entire "book." If it manages to find the correct position where the sentence is, shouldn't it also find the token "saudar"? Am I missing something here? Could it be because of the \n characters in the database? Honestly, I’m feeling completely lost.
rotund sonnet
Full text: Viso-rey amigo, eu
Sentence: Viso-rey da India amigo, eu
So when you do sentence_start = normalized_text.find(normalized_sentence)
This will won't find it, because it's not an exact match. There's extra characters in the sentence that prevent the full match against the text
The python .find requires an exact sub-string match to the larger string. You don't have that there
brittle nova
Hold on, I have more examples to share. Could it be that the matching process is being done poorly?
rotund sonnet
Your match_sentences function only requires a 90% rapidfuzz score to match, but your locate_token_in_text in its current form requires a perfect sub-string to text match
brittle nova
Can i send you the example through pm because its to long to be here?
rotund sonnet
I don't normally do DMs, but sure
