for loop completely breaks the purpose

[tame heath]

package main

import (
    "fmt"
    "github.com/gocolly/colly/v2"
    "log"
    "strings"
)

func url(c *colly.Collector) {

    for count := 0; count < 3; count++ {
        url := fmt.Sprintf("https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_pgn=%d&rt=nc", count) // Pagination
        fmt.Println(url)
        err := c.Visit(url)

        if err != nil {
            log.Println(err)
        }
    }
}

func main() {

    var titleArray []string
    var priceArray []string

    c := colly.NewCollector(colly.UserAgent("Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"))

    c.OnHTML(".s-item__title", func(element *colly.HTMLElement) {
        element.ChildAttr("heading", "role")
        titleArray = append(titleArray, element.Text)

    })

    c.OnHTML(".s-item__detail.s-item__detail--primary", func(element *colly.HTMLElement) {
        element.ChildText(".s-item__price")
        priceArray = append(priceArray, element.Text)

    })

    url(c)

    defer func() {
        for i := 0; i < len(titleArray); i++ {
            titles := strings.TrimSpace(titleArray[i])
            prices := strings.TrimSpace(priceArray[i])
            fmt.Printf("Title | %s | Price %s \n", titles, prices)

        }
    }()

}

With my current code, it prints all the URLs and THEN scrapes, so if I wanted to scrape all 109 pages I would have to wait multiple minutes for the URLs to print and only then it would start to scrape.

for count := 0; count < 3; count++ { // page count

In this example I change the page count to only go to 3. It then prints the 3 urls, once it has finished printing the URLs it starts to scrape.

What should happen? Scrape the page, print the url. Go onto the next one.

I've tried adding: https://pkg.go.dev/github.com/gocolly/colly/v2#Collector.Wait

and https://pkg.go.dev/sync#example-WaitGroup

but have had no success with either, anyone know how to fix it?

as shown in the screenshot, it prints all urls that it scrapes, instead of scrapin 1 page, printing the url and then going to the next

[autumn ice]

why are you using defer for the code that prints the price i see that's not the issue you are having with, the last defer is just as a debug view

i am guessing visit adds the url into the queue

where did you tried adding wait?

func url(c *colly.Collector) {

    for count := 0; count < 3; count++ {
        url := fmt.Sprintf("https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_pgn=%d&rt=nc", count) // Pagination
        fmt.Println(url)
        err := c.Visit(url)
        
        if err != nil {
            log.Println(err)
        }
        c.Wait()//we wait first before going to the next loop
    }
}

is it here?

[tame heath]

is that where I should place it?

will try

what's wrong with defer?

@autumn ice Still the same thing

[autumn ice]

what's wrong with that
does it not scrape the page?

where's the scraped stuff being posted i can only see

fmt.Println(url)

so i assume "scrape the page" means to just visit and do it's stuff

do you mean something else instead? like these?

these will only be printed when the program is exiting

it will always be the last part that gets printed

because your main function's code is effectively like this

func main() {
    var titleArray []string
    var priceArray []string

    c := colly.NewCollector(colly.UserAgent("Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"))

    c.OnHTML(".s-item__title", func(element *colly.HTMLElement)//...

    c.OnHTML(".s-item__detail.s-item__detail--primary", func(element *colly.HTMLElement)//...

    url(c)

    //the defer means nothing here since this is the last statement
    //that means these function only run after url(c) returns
        for i := 0; i < len(titleArray); i++ {
            titles := strings.TrimSpace(titleArray[i])
            prices := strings.TrimSpace(priceArray[i])
            fmt.Printf("Title | %s | Price %s \n", titles, prices)
        }
}

[tame heath]

ok so what I am expecting @autumn ice

Is for it to paste the URL

scrape the page

move onto the next page

repeat

not

print all pages, and then scrape them

[autumn ice]

what do you mean by "scrapes the page"

[tame heath]

title + price

[autumn ice]

is this what you mean when you say "scrape the page"

[tame heath]

yeah but

if I want to scrape all 109 pages

It isn't efficient

[autumn ice]

see what i said before

[tame heath]

ah, it's a problem with defer?

[autumn ice]

i have addressed what's the cause of your issue

[tame heath]

wtf

[autumn ice]

it has nothing to do with defer
see around here

[tame heath]

yes I read it

it still scrapes the page

but it prints the urls first, for some reason.

[autumn ice]

both images are the same right? (except for the red line)

[tame heath]

huh?

yeah

[autumn ice]

yes because the printing of the page is the last thing your program will do

[tame heath]

yep

[autumn ice]

if you dont want that, you could try moving the code that prints the page data into the url function inside the for loop

so that it prints the data every time the scraper is ran

[tame heath]

so put most of it in that function?

[autumn ice]

there's also the other way of registering an OnScraped handler on colly
https://pkg.go.dev/github.com/gocolly/colly/v2#Collector.OnScraped

yeah if you want it to post stuff after each page is done with being scraped

you might actually be better off registering as an OnScraped handler

so that you wont have to worry about concurrencies

[tame heath]

package main

import (
    "fmt"
    "github.com/gocolly/colly/v2"
    "log"
    "strings"
)

func url(c *colly.Collector) {

    for count := 0; count < 3; count++ {
        url := fmt.Sprintf("https://www.ebay.com/sch/i.html?_from=R40&_nkw=iPhone&_sacat=0&_pgn=%d&rt=nc", count) // Pagination

        err := c.Visit(url)

        if err != nil {
            log.Println(err)
        }
        c.Wait() //we wait first before going to the next loop
    }
}
func main() {

    var titleArray []string
    var priceArray []string

    c := colly.NewCollector(colly.UserAgent("Mozilla/5.0 (X11; Linux x86_64; rv:108.0) Gecko/20100101 Firefox/108.0"))

    c.OnHTML(".s-item__title", func(element *colly.HTMLElement) {
        element.ChildAttr("heading", "role")
        titleArray = append(titleArray, element.Text)

    })

    c.OnHTML(".s-item__detail.s-item__detail--primary", func(element *colly.HTMLElement) {
        element.ChildText(".s-item__price")
        priceArray = append(priceArray, element.Text)

    })

    url(c)

    for i := 0; i < len(titleArray); i++ {
        titles := strings.TrimSpace(titleArray[i])
        prices := strings.TrimSpace(priceArray[i])
        fmt.Printf("Title | %s | Price %s \n", titles, prices)
    }
}

Like even with

printing the url (I got rid of it)

it waits 10 seconds

and THEN

scrapes it

it's so weird

[autumn ice]

look here's how your code looks like when flatterned

func main{
//define stuff ...
  for i:=0; i<3;i++{
    c.Visit(url)
  }

  foreach of titleArray{
    ... //print the stuff
  }

you are only printing the stuff ONCE

and you print it AFTER the function url(c) returns

[tame heath]

hmmm

yeah I understand

[autumn ice]

so basically the program performed as per instructed with your given code

nothing is weird about it here

[tame heath]

can you put it into the existing code?

I understand the flattening

ah

[autumn ice]

you just have to move the code where you print the list, into the for loop

i cant i am outta time and i gotta do some other stuff

[tame heath]

so the for loop?

[autumn ice]

you should move it(the code that prints the price) into the loop

[tame heath]

yeah but

what about the

if I delete the existing var titlearray and price array

I can't append anything

[autumn ice]

yep you would have to restructure it
OR you can try using c.OnScraped() instead

https://pkg.go.dev/github.com/gocolly/colly/v2#Collector.OnScraped

that might be easier

[tame heath]

yeah I think so lol let me try

[autumn ice]

you just need to provide another callback which will be called when page has been crawled

[tame heath]

where should I use the onscraped?

new func?

[autumn ice]

func (c *Collector) OnScraped(f ScrapedCallback)

you need to pass it a function as callback

[tame heath]

yeah I tried that

but it turned red lol

with red line

[autumn ice]

you cant copy paste it

you need to pass a function that complies with the signature

which means you call c.OnScraped(...)

this is the actual type of ScrapedCallback
https://pkg.go.dev/github.com/gocolly/colly/v2#ScrapedCallback

the rest is left as an exercise to the reader

i am not here to hand hold you thru everything

[tame heath]

onScraped doesn't solve the issue.

[autumn ice]

i see, sadly i am out of time then

you will have to wait for someone else, good luck

[tame heath]

all g

[autumn ice]

you should still consider posting what you tried and why it didnt work for the next person

[tame heath]

don't think anyone has had the same issu

been trying for ages to fix this

[autumn ice]

well maybe that's because it's an issue with your code

i think my explanation had made it very clear code is performing as instructed

[tame heath]

it's not performing as expected, at all. It's absolutely inefficient.

[autumn ice]

*instructed, per the manner you have written it

[tame heath]

well, yeah. lol

[autumn ice]

you still havent tell others what you tried btw

which will make it inefficient for others to try to help you

[tame heath]

I tried everything you told me

[autumn ice]

because they will have to ask you what you did

like for example how does it not solve it

what's the new code you used

[tame heath]

how can I explain it more?

I literally said what's wrong with the code and what I intent for it to do

[autumn ice]

the new code you tried i guess?

you said you tried using onScraped and it didnt solve it
but you did not show the code you use to try it

without knowing the steps you took, no one else can know if the steps you took are correct, or it's an actual fundamental issue

that's what i mean when i said not enough information for others to assist

[tame heath]

I'm sorry, but I definitely gave enough information.

and an expectation

[autumn ice]

if that's what you mean when you said the new code you tried
that's just doing it incorrectly

[tame heath]

no, that's not the new code I tried.

[autumn ice]

i see, so that new code you tried is not shared

fair enough

[tame heath]

idk

[autumn ice]

i think i said everything i could have said, i dont have any other ideas to help you
sorry, hopefully someone else can help you with what information was given

[tame heath]

yea all g ty