Web scraping tool

Russinating · 9 May 2022 at 11:19

I need to scrape a couple of webpages for data that's publicly accessible (tho it's behind a login, if that matters).

Both are just a list of profiles. The main page has a list of everyone's names, and their subsequent/individual page has further details they've opted to share. So I need a scraper that will follow each link and export the data to a spreadsheet. I've tried using a couple but they don't have the "follow link" functionality and so they only scrape the names.

Any recommendations? I can write PHP and can run something locally if that helps.

Hades · 9 May 2022 at 21:53

I would probably start by taking a look at Python with one of the web scrapin packages such as Beautiful Soup.

Russinating · 11 May 2022 at 10:12

Hades said:
I would probably start by taking a look at Python with one of the web scrapin packages such as Beautiful Soup.

Yeah thought that'd be a response. I'm not familiar with Python and given it's for business I've just paid someone on Fiverr to do it.

DarrenM343 · 13 May 2022 at 10:46

A fiver? Where are these people? Maybe I can outsource all my freelance work to them.

Another option would be any popular language combined with selenium. It will drive a browser to do it but think can use it in headless mode. Easy to use, can use browser F12 tools to help find elements by class name or id etc

Dj_Jestar · 17 May 2022 at 00:01

@DarrenM343 not a fiver, on Fiverr

Russinating · 17 May 2022 at 19:11

Dj_Jestar said:
@DarrenM343 not a fiver, on Fiverr

Doh! Anyway the data came back great in less than 24 hours.

Lmg80 · 28 May 2022 at 21:18

DarrenM343 said:
A fiver? Where are these people? Maybe I can outsource all my freelance work to them.

Another option would be any popular language combined with selenium. It will drive a browser to do it but think can use it in headless mode. Easy to use, can use browser F12 tools to help find elements by class name or id etc

Selenium automates web navigation, not parsing / scraping, unless the op is after embedded attachments to download.

Op as mentioned beautiful soup is the way forward, and even for a newbie is intuitive / Google as you build out what you need. Plenty of help on Stackoverflow