PHP cURL with authentication

AfterEffect · 10 Jun 2012 at 23:54

Hey all,

I'm trying to scrape the activity feed from the Barclay's Cycle Hire website but have come to a dead end because the page with the login form seems to set a hidden POST variable, so I'll need to retrieve and cache that in cURL, but not sure how

The login form is here:
https://web.barclayscyclehire.tfl.gov.uk/

It seems to submit three POST variables:

login[Username]
login[Password]
login[_csrf_token]

The csrf_token one seems completely random each time you load the page... so not sure how to do this in cURL

edit:
Basic form code:

Code:

<form action="/" method="post">
       <fieldset>
                        <label for="login_Email">Email address</label>                        <input autocomplete="off" size="40" maxlength="254" type="text" name="login[Email]" id="login_Email" />
                        <label for="login_Password">Password</label>                        <input autocomplete="off" size="20" maxlength="16" type="password" name="login[Password]" id="login_Password" />
                        <!-- SUBMIT BUTTON -->
                        <input type="submit" value="Sign in" class="button-default-1" />
            <!-- TOKEN -->
                        <input type="hidden" name="login[_csrf_token]" value="a213399200aa6fb6d90e3de70835d23d" id="login__csrf_token" />       </fieldset>
	</form>

edit2!: I tried using pret_match to grab the value but I think it gets a different value when cURL connects to the site

visibleman · 11 Jun 2012 at 14:38

Have you tried grabbing the token with one CURL request and sending it back (with user + pwd) on another?

I'd have a look at the headers on the first request, especially for anything session based, and would send those back (use the --header/-h parameter) along with the 'login' request.

Also have a go at using Fiddler, it might help you to see what data is being swapped when you go to the page and login.

rickh · 11 Jun 2012 at 15:56

Once you understand what the _csrf_token is you'll understand the problem.

http://en.wikipedia.org/wiki/Cross-site_request_forgery
http://stackoverflow.com/questions/6929449/are-we-really-secured-from-csrf

AfterEffect · 13 Jun 2012 at 19:53

Hmm, so it's pretty much impossible?

If you use a wget to grab the data then the next wget to post the data creates a new csrf token :/

AfterEffect · 30 Aug 2012 at 13:01

Really sorry to bump an old thread, but sort of gave up based on the above advice, however I've just seen somebody has released an android app which does what I'm trying to do... so there must be a way!!

I've played around with cURL but haven't used it for any advanced stuff before so not sure what to do about the csrf token

Any help would be HUGELY appreciated

visibleman · 30 Aug 2012 at 13:22

You sure there isn't an API of sorts?

Edit - Looks like there are a handful of API's available - http://www.tfl.gov.uk/businessandpartners/syndication/default.aspx

AfterEffect · 30 Aug 2012 at 13:42

visibleman said:
You sure there isn't an API of sorts?

Edit - Looks like there are a handful of API's available - http://www.tfl.gov.uk/businessandpartners/syndication/default.aspx

There are for bike availability but not for specific account history. You can only get that by logging into the Cycle part of the TfL site.

visibleman · 30 Aug 2012 at 14:10

Well the [_csrf_token] isn't completely random on each page load, rather it changes depending on if a cookie is stored or not. So have a look at the CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options of CURL and see if that works.
You're still going to have to call CURL twice - once for grabbing the token and setting the cookie, another to send the token/cookie and username/password.