What are you using to scrape? I used Python last year to scrape off hotel rooms and prices for a site to book them. The original website was very javascript/ajax heavy, but I managed to get around it with Selenium to wait for certain things to appear and click certain buttons. Was a fun thing to do.
web scraping - general advice, tips etc..
- Thread starter dowie
- Start date
Soldato
Copyright/legal/moral/ethical issues aside:
- Run the scrape during a period when traffic on the site would be quiet e.g 2am if its a UK site.
- Scrape slow. Pulling a page every two seconds is 30 a minute, 1800 in an hour. For the volume you indicated, its 2-3 hours to get the lot.
- Scrape smart. How often are the pages likely to change. Do you need a daily run ? Will they also change at the weekend. Will they change during public holidays.
There is no 'best' language or tool, its whatever you feel comfortable working with. You can google code for scrapers in every language going, its a classic tutorial programming assignment.
- Run the scrape during a period when traffic on the site would be quiet e.g 2am if its a UK site.
- Scrape slow. Pulling a page every two seconds is 30 a minute, 1800 in an hour. For the volume you indicated, its 2-3 hours to get the lot.
- Scrape smart. How often are the pages likely to change. Do you need a daily run ? Will they also change at the weekend. Will they change during public holidays.
There is no 'best' language or tool, its whatever you feel comfortable working with. You can google code for scrapers in every language going, its a classic tutorial programming assignment.
Caporegime
Web scraping without permission is illegal. Hardly off topic, and certainly not drivel.
I was waiting for you to repeat that line about the CJEU's ruling, because on the very next paragraph you've failed to read:
The reading failure isn't mine, you're going round in circles - you've made an assertion you can't back up re: illegality, you've made claims re: copyright that your case shows to be untrue - the case related to breach of contract and was won by Ryan air as to use their website the company concerned had to agree to terms and conditions which they then broke, I've already replied to this - please note the last part of my reply. If you still don't understand then I'd rather you started another thread instead of carrying on repeating the same nonsense in here.
Last edited:
thanks for the on topic reply
I was looking at making use of BeautifulSoup with python - any libraries you'd recommend?
thanks, there do seem to be quite a few libraries out there - it might be the case that scraping even 2 - 3 days is workable instead. I'm unsure about weekends/public holidays - possibly not updated then but I'll need to test that.
The site is likely quite busy and able to handle a lot of traffic so re: your 2am suggestion, as I'm unlikely to be causing much issue for them re: traffic, would that change things re: your advice there - would scraping at a time with less traffic mean I'm more likely to be detected in logs etc..?
Soldato
The nail that sticks up is the one that gets hit. Do anything you can to minimise being that nail.
Why not just a quick email asking if they mind you scraping some information from their site? You don't necessarily have to say up front what you'd be using it for. If you're going to be doing it in the long term it seems likely they'd notice eventually. All this talk of being discrete or that they might not be happy suggests you perhaps shouldn't be doing it otherwise.
What do the T&Cs say?
I think in general most websites aren't necessarily open to scraping(at least I'd assume so?) - though like I said before I'd want to be discrete about this, I can't really expand much more regarding that other than to say approaching them isn't going to be useful in that respect - so even if they'd be welcoming it doesn't really make much difference in this instance
nothing in the T&Cs about web scraping so the issue previously highlighted re: the Ryan Air case doesn't apply
nothing in the T&Cs about web scraping so the issue previously highlighted re: the Ryan Air case doesn't apply
Possibly if the site is not ajax heavy or requires automated clicking. My web scraper had to enter lots of stuff into fields, click buttons, navigate the site etc to get data, so Selenium was best for that as it has a lot of nice features.
Is there any common structure to the data on the pages you are wanting to scrape, and where to you intend to deposit (and in what format) the data once you have it?
E.g:
- "I want to scrape all headings", or "I want to scrape a paragraph below a particular heading". These are easy if they are consistent. If the question is more.. "I want to run some kind of logic to understand which part of the page to scrape, then I need to format it based on some rules", then it's more difficult.
You want to run it once per day - do you have your own server, or are you leaving a cron job (or similar) running locally?
For the amount it would cost (peanuts), i'd recommend running a aws lambda function daily. This can be written in Node.js, Java, C# or Python. If it was me, as I work mainly with Java, i'd do it in Java... but all of those languages can easily request web pages and parse them. Using Java, i'd recommend JSoup. I don't know about the others. From there it's easy to store in S3, or in a database.
Regarding the legality, i'd be very careful. I suspected it may not be legal (I'm no legal expert), but some quick googling comes up with things like:
"Most courts are in agreement that web scraping is unlawful, but that does not mean that all web scrapers are identified and punished."
"In 2001, the legality question was brought up again when a travel agency sued another company for scraping information. The rival company made use of the pricing information taken, without consent and undercut the competition – and resulted in less customers and income for that agency. This brought to attention the importance of authorized and unauthorized access of information on websites and how to ensure that no unauthorized users could scrape information."
https://www.scrapesentry.com/scraping-wiki/web-scraping-legal-or-illegal/
So yes, I would imagine it is illegal to scrape OcUKs prices and then use them to undercut them - but i'm not an expert in the matter. I just know it seems to be a contested subject.
E.g:
- "I want to scrape all headings", or "I want to scrape a paragraph below a particular heading". These are easy if they are consistent. If the question is more.. "I want to run some kind of logic to understand which part of the page to scrape, then I need to format it based on some rules", then it's more difficult.
You want to run it once per day - do you have your own server, or are you leaving a cron job (or similar) running locally?
For the amount it would cost (peanuts), i'd recommend running a aws lambda function daily. This can be written in Node.js, Java, C# or Python. If it was me, as I work mainly with Java, i'd do it in Java... but all of those languages can easily request web pages and parse them. Using Java, i'd recommend JSoup. I don't know about the others. From there it's easy to store in S3, or in a database.
Regarding the legality, i'd be very careful. I suspected it may not be legal (I'm no legal expert), but some quick googling comes up with things like:
"Most courts are in agreement that web scraping is unlawful, but that does not mean that all web scrapers are identified and punished."
"In 2001, the legality question was brought up again when a travel agency sued another company for scraping information. The rival company made use of the pricing information taken, without consent and undercut the competition – and resulted in less customers and income for that agency. This brought to attention the importance of authorized and unauthorized access of information on websites and how to ensure that no unauthorized users could scrape information."
https://www.scrapesentry.com/scraping-wiki/web-scraping-legal-or-illegal/
So yes, I would imagine it is illegal to scrape OcUKs prices and then use them to undercut them - but i'm not an expert in the matter. I just know it seems to be a contested subject.
Soldato
- Joined
- 19 Oct 2008
- Posts
- 5,994
Personally think some of you guys are taking this too seriously without knowing what the info/data is. Sounds to me like OP wants something like stock prices. Some sites actually allow you to download such information as csv's. If OP is using the information for himself only I don't see a problem with it, they're not going to be passing it off as their own work, or giving it to others. And it sound like the data is freely available anyway, ie, not like music where you have to pay for it, not a subscription service.
Of course the company might not like the artifical load being placed on their servers, but that's a different matter.
It's actually pretty easy to scrape info.
worth noting that there are engines out there that will search the net for text to help with decision, such as the buying of stocks. The engines will find news articles about companies. I have no idea if they help however but it's in use and is automation of what a person could do., just much quicker. I suppose these engines parse data rather than scrape it though.
I'm not saying it's right though but really depends what it is they're intending to scrape.
Of course the company might not like the artifical load being placed on their servers, but that's a different matter.
It's actually pretty easy to scrape info.
worth noting that there are engines out there that will search the net for text to help with decision, such as the buying of stocks. The engines will find news articles about companies. I have no idea if they help however but it's in use and is automation of what a person could do., just much quicker. I suppose these engines parse data rather than scrape it though.
I'm not saying it's right though but really depends what it is they're intending to scrape.
Last edited:
Just to update this US courts have affirmed recently that web scraping is legal.
techcrunch.com
This wasn't (directly) about stock prices (that data is easily available from a range of providers/doesn't require scraping) but rather was data tangentially related to stock prices, incidentally over the past 5 years the use of "alternative data" has become a bit more widespread.
Web scraping is legal, US appeals court reaffirms | TechCrunch
The landmark web scraping case was bounced back to the Ninth Circuit by the U.S. Supreme Court.
techcrunch.com
This wasn't (directly) about stock prices (that data is easily available from a range of providers/doesn't require scraping) but rather was data tangentially related to stock prices, incidentally over the past 5 years the use of "alternative data" has become a bit more widespread.
Soldato
- Joined
- 16 Aug 2009
- Posts
- 7,947
It might be legal (and thats a US ruling) but that doesn't mean the site owner has to like it - I've known people get IP banned for scraping sites without permission, always ask first.