Dark SEO Programming
SEO Tools. I code ‘em.

If you looked over at the guest post code on BlueHatSeo.com for scraping it’s a little incomplete (in my opinion ). It’s missing cookies, and has a couple of flexibility issues. The code below will let you use POST variables simply, as well as allowing you to store session data in cookies etc.

It’s really simple to use. To get a page without proxy and no post variables:

$htmlcode = scrape_page(”http://www.google.com/”);

with post variables:

$htmlcode = scrape_page(”http://www.google.com/”, 1, “var1=1&var2=2&var3=3″);

with proxy (defaults to 127.0.0.1:8118 - TOR):

$htmlcode = scrape_page(”http://www.google.com/”, 0, “”, 1);

Here’s the code… all you need to do is change the cookie path to a text file (with the correct permissions on linux), and set the proxy to your proxy address.

<?php

function scrape_page($page, $post=0, $fields=null, $proxy=0)
{
// cookie path
$file_cookie = “/path/to/cookie/file/cookies”;

$ch = curl_init($page);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $file_cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $file_cookie);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

if($proxy==1)
curl_setopt($ch, CURLOPT_PROXY, “127.0.0.1:8118″);

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_USERAGENT,
“Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)”);

if($post==1)
{
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
}

$response = curl_exec($ch);
curl_close($ch);

//echo curl_error($ch);

return $response;
}

?>

The only thing it is missing is some good old error checking.

This entry was posted on Tuesday, March 11th, 2008 at 5:57 am and is filed under mini script, scraper. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.