Go Colly, how to select a paragraph within a div | Discord Gophers | Page 1

vale kernel Nov 10, 2022, 7:42 AM

#

I am trying to select a small piece of text within a paragraph but I am unable to get it working.
I tried finding a tutorial online but nothing helped so far.

Hope someone can shine a light on what might be the solution.

<div class="priceindicator__container indicator-block"><svg class="priceindicator__svg"><use xlink:href="/images/svg/svg.svg#icon-daypart-ceremony"></use></svg> <div class="priceindicator__description"><h4> TITLE_HERE </h4> <p> TEXT_HERE <a class="text--s s-hide m-hide" tabindex="0"> <span id="tooltip-wrapper"> <svg id="1" class="preload-fas-fa-question-circle"><use xlink:href="#preload-fas-fa-question-circle"></use></svg> </span> </a> </p> </div> </div>

I want the piece of text that is within the "p" inside of the div with the "priceindicator__description" class. Hope someone can explain to me how to do it.

c.OnHTML(".priceindicator__description p", func(e *colly.HTMLElement) {
//a := e.DOM.Find("p")
//fmt.Println(a.Html())
// fmt.Println(e.ChildText("p"))
})

This is what I tried in various ways, but nothing is giving me the text within the P tag.

buoyant relic Nov 10, 2022, 8:08 AM

#

package main

import (
    "fmt"

    "github.com/gocolly/colly"
)

func main() {
    c := colly.NewCollector()

    c.OnHTML(".priceindicator__description p", func(e *colly.HTMLElement) {
        fmt.Println(e.DOM.Text())
    })

    if err := c.Visit("https://dozn.net/colly.html"); err != nil {
        panic(err)
    }
}

#

You can use strings.TrimSpace() to remove the extra spaces if you have that in the original HTML as well

vale kernel Nov 10, 2022, 8:36 AM

#

Ohhh you are a hero Dozn, been trying to get it working for a whole day lol. Thank you very much, will try it tonight and let you know!

buoyant relic Nov 10, 2022, 8:36 AM

#

No problem ^_^

vale kernel Nov 10, 2022, 9:22 AM

#

Ahw, it returns nothing 😦

#

https://www.theperfectwedding.nl/bedrijven/9673/on-the-rock is the URL I am trying to do this magic on. The text I would like to extract is 'v.a. € 12,00 p.p.'

buoyant relic Nov 10, 2022, 9:25 AM

#

It loads all of that using JavaScript

#

vale kernel Nov 10, 2022, 9:25 AM

#

Ohhhh dang and colly doesn't handle that properly? that means I will need a headless option?

buoyant relic Nov 10, 2022, 9:25 AM

#

#

Depends on how they do it, sec

#

Still looking...

vale kernel Nov 10, 2022, 9:35 AM

#

No worries take your time, really appreciating your help!

paper valley Nov 10, 2022, 9:36 AM

#

i've been using surf for things like this so if colly doesn't support it I highly recommend surf

#

https://surf.readthedocs.io/

buoyant relic Nov 10, 2022, 9:37 AM

#

Yeah there's a few for using an actual browser, but you generally want to avoid using them since they're slow

vale kernel Nov 10, 2022, 9:40 AM

#

Surf? oh good to know thanks Dr. Drip 🙂 But yeah slow, I need to scrape a couple of thousand pages soooo.... goopturtle

paper valley Nov 10, 2022, 9:41 AM

#

hahah fair enough

buoyant relic Nov 10, 2022, 9:41 AM

#

Can't believe it's this hard to find the price rofl

#

Normally it's just an XHR request, or it's embedded in the HTML. Not in this case, so I'm scouring the JS

vale kernel Nov 10, 2022, 9:44 AM

#

Hahaha, yeah I had a hard time too. I can use a headless browser but I figured it was my zero experience with Colly that was the issue 😄

buoyant relic Nov 10, 2022, 9:44 AM

#

I've made that mistake a few times in the past where I didn't realize it was being loaded dynamically, so I know the feeling 😉

buoyant relic Nov 10, 2022, 10:24 AM

#

I think they just obfuscated everything so you can't read it =/ Don't think I'll find anything at this rate lol

#

https://tenor.com/view/shiba-inu-attempt-tricks-fail-trick-gif-4466269

Tenor

vale kernel Nov 10, 2022, 10:26 AM

#

hahahaha

#

No problem, thanks so much for your time and help 🙂

#

Think ill just try out Surf as Dr. Drip suggested

buoyant relic Nov 10, 2022, 10:27 AM

#

No problem, best of luck

vale kernel Nov 10, 2022, 10:27 AM

#

Javascript websites are awesome but not when you want to crawl them 😄

buoyant relic Nov 10, 2022, 10:27 AM

#

Can't wait for WASM too

#

Where you'll have to crawl through the assembly

#

shudders

#

Or rather, WASM+rendering using graphics instead of the DOM

vale kernel Nov 10, 2022, 10:29 AM

#

hehehe, will be joyful days 🤪

buoyant relic Nov 10, 2022, 10:40 AM

#

Holy shit I figured it out

#

{
    "id": 3,
    "category_id": 17,
    "name": "Feest",
    "type": "number",
    "use_in_avg": 1,
    "default": 50,
    "min": 0,
    "max": 0,
    "question": "Wat is de minimale prijs van een trouwfeest met 50 gasten?",
    "description": "Ga hierbij uit van 4 uur feest met 50 personen, inclusief drank en minimale kosten voor zaalhuur.",
    "frontend_question": "",
    "metric": "p.p.",
    "priority": 0,
    "frontend_description": null,
    "icon": "icon-daypart-party",
    "factor": 50,
    "tag_id": 851,
    "profile_id": 9673,
    "avg": 1775,
    "tag_set": true
}

#

avg / factor = price

#

So this one's for 35,50

#

It's directly in the HTML

#

package main

import (
    "encoding/json"
    "fmt"

    "github.com/gocolly/colly"
)

type Items []struct {
    Name                string      `json:"name"`
    Question            string      `json:"question"`
    Description         string      `json:"description"`
    FrontendQuestion    string      `json:"frontend_question"`
    FrontendDescription interface{} `json:"frontend_description"`
    Factor              int         `json:"factor"`
    Avg                 int         `json:"avg"`
}

func main() {
    c := colly.NewCollector()

    c.OnHTML("w-profile-price-indicator", func(e *colly.HTMLElement) {
        items := Items{}
        priceIndicators := e.Attr(":price-indicators")
        if err := json.Unmarshal([]byte(priceIndicators), &items); err != nil {
            panic(err)
        }

        for _, item := range items {
            price := float64(item.Avg) / float64(item.Factor)
            fmt.Printf("%s: %.2f\n", item.Name, price)
        }
    })

    if err := c.Visit("https://www.theperfectwedding.nl/bedrijven/9673/on-the-rock"); err != nil {
        panic(err)
    }
}

#

Output: ```
Ceremonie: 12.00
Receptie: 10.00
Feest: 35.50
Diner: 64.00
Overnachten: 0.00

buoyant relic Nov 10, 2022, 11:10 AM

#

Worst part is that's the first place I looked, but couldn't connect the values >_<

vale kernel Nov 10, 2022, 3:44 PM

#

sorry wasn' t there, but OMG thats awesome! let me read it a bit haha

buoyant relic Nov 10, 2022, 3:44 PM

#

np

vale kernel Nov 10, 2022, 3:44 PM

#

ohhh wow super nice!!!

buoyant relic Nov 10, 2022, 3:44 PM

#

Enjoy ^_^

vale kernel Nov 10, 2022, 3:44 PM

#

HERO! 😄

vale kernel Nov 10, 2022, 4:02 PM

#

Works perfect indeed, thank you so much!!! now just a couple more things and done hahaha. Don' t know why but I am having a hard time using Colly haha, not sure when to use what function et cetera 😄

#

strings.TrimSpace(content.Text()) thats nice 😄

#Go Colly, how to select a paragraph within a div