🔒 Help with parsing and structuring data

[bronze pecan]

If I have a JSON of scraped listing of laptops, but the descriptions were messy, how can I reliably extract specs, model name, CPU etc from the listing?

[pearl loom]

@bronze pecan

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[bronze pecan]

Example data:

[pearl loom]

Click here to see this code in our pastebin.

[bronze pecan]

It must also delete listings which have insufficient information

[clear osprey]

!e You could do something like this:

import re

description = "Apple MacBook Air 13.3\" (2017) – i5 – 8GB RAM – 256GB SSD – inkl. OVP & Original Ladegerät✅ Generalüberholt & sofort einsatzbereitZuverlässiges und leichtes Apple MacBook Air – ideal für Studium, Office, Surfen & Alltag.✔ Generalüberholt & vollständig geprüft✔ macOS Monterey frisch installiert✔ Sofort startklar – kein Setup nötig✔ 1 Jahr Gewährleistung✔ Rechnung inklusive✔ Original Apple Ladegerät inklusive✔ Originalverpackung (OVP) enthalten⚡ Leistungsstarke AusstattungIntel Core i5 Dual-Core (1,8 GHz)8GB RAM256GB SSD (schnell & zuverlässig)Intel HD Graphics 6000️ Display & Design13,3 Zoll Display1440 x 900 Pixelkompakt, leicht & ideal für unterwegsAnschlüsse & KonnektivitätThunderbolt 2 (Mini DisplayPort)2x USB 3.0SDXC KartenleserMagSafe 2 LadeanschlussWLAN (ac) & Bluetooth 4.0Akku✔ Zustand: Gut (laut Apple System)solide Laufzeit für den "

cpu = re.search(r"(i5|i3|i7)", description)
if cpu: print(cpu.group(0))

ram = re.search(r"(\d+\s*GB)\s*RAM", description)
if ram: print(ram.group(0))

ssd = re.search(r"(\d+\s*GB)\s*SSD", description)
if ssd: print(ssd.group(0))

[pearl loom]

:white_check_mark: Your 3.14 eval job has completed with return code 0.

001 | i5
002 | 8GB RAM
003 | 256GB SSD

[clear osprey]

I notice that each section is also separated by a special character, so maybe you could split on – first, and then do checks.

!e like this:

description = "Apple MacBook Air 13.3\" (2017) – i5 – 8GB RAM – 256GB SSD – inkl. OVP & Original Ladegerät✅ Generalüberholt & sofort einsatzbereitZuverlässiges und leichtes Apple MacBook Air – ideal für Studium, Office, Surfen & Alltag.✔ Generalüberholt & vollständig geprüft✔ macOS Monterey frisch installiert✔ Sofort startklar – kein Setup nötig✔ 1 Jahr Gewährleistung✔ Rechnung inklusive✔ Original Apple Ladegerät inklusive✔ Originalverpackung (OVP) enthalten⚡ Leistungsstarke AusstattungIntel Core i5 Dual-Core (1,8 GHz)8GB RAM256GB SSD (schnell & zuverlässig)Intel HD Graphics 6000️ Display & Design13,3 Zoll Display1440 x 900 Pixelkompakt, leicht & ideal für unterwegsAnschlüsse & KonnektivitätThunderbolt 2 (Mini DisplayPort)2x USB 3.0SDXC KartenleserMagSafe 2 LadeanschlussWLAN (ac) & Bluetooth 4.0Akku✔ Zustand: Gut (laut Apple System)solide Laufzeit für den "

parts = description.split(' – ')
print(*parts[:4], sep='\n')

[pearl loom]

:white_check_mark: Your 3.14 eval job has completed with return code 0.

001 | Apple MacBook Air 13.3" (2017)
002 | i5
003 | 8GB RAM
004 | 256GB SSD

[clear osprey]

@bronze pecan ?

[bronze pecan]

The problem is, this is scraped from a public listing of classifieds. So the titles aren't always consistent.

1. How can I fallback to the description if the titles aren't consistent
2. How do I infer the model name for each laptop, like MacBook pro M1, dell Inspiron 7xxxx, LG notebook g10 etc etc

I want to be able to get the exact model name, screen size (if provided), ram, GPU, CPU, storage, model year and write it all into a JSON file

[ionic thistle]

hello

[short patrol]

why not just scrape them as is? skip the parse?

[ionic thistle]

neither will work, unless you have a RAG with a database of every model of laptop that has ever been made

[bronze pecan]

Well I need to parse

That's a part of my project

[bronze pecan]

Is it possible to get that?

I know how to build RAG

And how it works

And I don't need every listing of every laptop ever made

[ionic thistle]

there is no way you can get clean data out of a dataset that is so inconsistent, because at some point you are going to run into titles that even humans cannot find the model for

[bronze pecan]

Just ones > 2015

[short patrol]

well you've mentioned the problem. Its not consistently reliable to detect a pattern on how people enters their title and description into their ad.

[bronze pecan]

I can filter out low quality entries or fall back to description

[ionic thistle]

the issue is not filtering them, the issue is how do you determine if it's low quality

at some point even a human won't be able to tell

[brisk salmon]

depending on how many listings you've got (and if you only have to do it once), classifying them manually is also an option... It is painful, but I've done it before for stuff like this

[bronze pecan]

For example something called "Schönes (beautiful) Acer laptop" have to be filtered out automatically

Cause it doesn't mention model name

And only the brand name

I was thinking of keeping a hierarchy list

[ionic thistle]

examples cannot be used as a filter, can it?

[bronze pecan]

No, I have probably tens of thousands

Can't do it manually

[ionic thistle]

imagine if i had a laptop listing for "Acer 4736G"

i misspell it, and i get "Acer 4737G", a model which does not exist

does it filter it out?

[bronze pecan]

Maybe I have to research into naming conventions of certain laptop brands and then develop a regex expression for those?

[brisk salmon]

lol

[ionic thistle]

the problem of course, is that these companies are FAMOUS for having terrible and inconsistent naming conventions

[bronze pecan]

Yes it does

[ionic thistle]

so you only want the ones that are not misspelled?

[bronze pecan]

That's a low quality entry

[bronze pecan]

Yup

Or ones that actually exist

[ionic thistle]

how about small typos like "Ac er"

[bronze pecan]

Maybe I can take that into account for the regex

It'll accept a cer, ac er, or ace r

[ionic thistle]

even then, how do you tell if the model number is correct to the specifications

[bronze pecan]

What do you mean?

[ionic thistle]

what if i were to show up with a macbook air 11 with an AMD 9800x3d

macbook air 11 exists

AMD 9800x3d exists

[bronze pecan]

That's a low quality entry, instantly deleted

[ionic thistle]

how do you tell?

where did you get this info from?

[bronze pecan]

Macbooks must have only intel or M series chips

[ionic thistle]

yea, how did you know this?

you as a human know this

but how does the computer tell?

[bronze pecan]

I'll write that into the logic?

[ionic thistle]

how about for other laptop manufacturers?

[bronze pecan]

It's such a minute possibility that a listing will have incorrect GPU/CPU that I'm okay with that being wrong

That would account for maybe 1%

[ionic thistle]

was there ever HP G32 with an Intel 1145G7?

[ionic thistle]

nooooooooooo

you underestimate how much humans misspell model names

[bronze pecan]

I don't want it to be perfect, just good enough that it can account for 90% at least

[ionic thistle]

i know

but to get to 90% is already a problem in of itself

[bronze pecan]

See, most of them just type i5 or i3, or AMD Ryzen 7

Etc

Simple names

[ionic thistle]

hmm

[bronze pecan]

It's hard to get them incorrect

[ionic thistle]

you can regex for the common stuff like i3, i5, i7, ryzen 5, ryzen 7 etc

[bronze pecan]

Yes I know

I have that down

The only problem is the model names

[ionic thistle]

hmm

[bronze pecan]

I can't seem to find a good method for extracting that

Do you have any ideas 🥲

[ionic thistle]

checking

[ionic thistle]

you may need to build up a database of models from the respective manufacturers websites

you can probably get to 90% with just the major laptop manufacturers

[bronze pecan]

Manually?

[ionic thistle]

acer, asus, hp, dell, etc

[ionic thistle]

there may be automated ways, but i would prefer not to break rule 5

[bronze pecan]

Oh okay I get what you mean

[ionic thistle]

i dont know if this will work, and i can't help with that

[bronze pecan]

So I get all the model names and then maintain a herirarchy table

[ionic thistle]

this is probably what i would do

[bronze pecan]

And check if those keywords appear in the title/description

And assign the model_namw accordingly?

[ionic thistle]

after checking for manufacturer first

because this dataset is so inconsistent, i would not be able to tell the effectiveness of this method

with this database of model names, you can use a RAG but obviously regex will be WAY faster

and accurate

[bronze pecan]

Hmm

[short patrol]

Alternatively, could use Amazon Product Search API, refine the search down to just Electronics > Computers, and fetch all entered keywords and descriptions and specs from there...

[bronze pecan]

If I can find documents on the internet that has model names of most major laptop companies

Or collect them

Is it better to use RAG?

[bronze pecan]

And try to match the titles and get a confidence rating

I think that's a good idea too

[short patrol]

Better collect them from source somehow. Each brand should have some sort of product API exposed.

But I found something that you can start keeping track. Its a bit dated (3 years ago) but its better than nothing. https://www.kaggle.com/datasets/durgeshrao9993/laptop-specification-dataset

[bronze pecan]

Hmm thanks a lot! It's really useful

[short patrol]

Like for instance, if you're looking at Dell Laptops, they do have a listing on their page for you to try and scrape from: https://www.dell.com/en-us/shop/dell-laptops/scr/laptops

At least referring from the direct source, its a lot more up to date than getting from some intermediary API

[bronze pecan]

Problem is, most laptop companies will scrub discontinued models from their websites which would be listed in classified ads

I think I should have multiple fallbacks

[short patrol]

you can worry about discontinued models later. Because eventually, all of them will be discontinued. So you might as well get the latest and current now, and worry about finding and filling in the blanks for discontinued models later

[bronze pecan]

Hmm true

[short patrol]

fetching for missing info from current discontinued models will likely be manually be done. A quick google here and there, update it in your DB directly and thats it.

[pearl loom]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.