Google PageRank Checksum Algorithm

[medivh]

Hi,

I wonder if anyone found a solution to the PR calculation not working on big endian systems? i.e. a script code that works with 64 bit integers? would be glad if someone could help... ;-)

--chris

[chemoul]

Hi,

I have a strange problem, where I use a php script (the one in this thread) to check pagerank. It works on one of my servers, but on the other one I get the "forbidden" error from Google.
Any idea why?

Steve

[chemoul]

Also the checksums on both servers are different for one and the same domain. I have no idea why?

[Jippi]

What was the conculusion... was there a working php script ? that works on php 5.1.2 also ? been googling like crazy for the last days to find one, but none of them return the right ch's

[alexanderg]

Originally Posted by AboutSledge

Google PageRank Checksum Algorithm(GoogleToolbar 3.0.125.1-big)

http://www.gamesaga.net/pagerank.php

C and PHP implelation

http://ww.gamesaga.net/pr.zip
http://ww.gamesaga.net/pr.tar.gz

Url http://www.gamesaga.net/pagerank.php is not valid and site http://www.gamesaga.net it seems does not contain information about PageRank anymore.

Do you know another resources with description of the new GooglePR algorithm version 3+ ?

[Gagget]

Hello @all

At the beginnng i have to say that i'm a german guy and my english is not very well. In the last Days i worked on some changes for a Friend and his Installation of phpLinkDirectory (The Pagerank script is included there).

After I uploaded it on the Server (64bit Linux PHP 5.0.5), it appears exact the same problem with the different or false Google checksum ($sh). So I think NOT that PHP 5.1.2 is the Problem. I think the "Unregistred" is right. The maximum signed integer value for 64 bit systems is the Problem.

Tonight, after several Hours of work, I found a Solution, but i couldnt test it on a Server with PHP 5.1.2 and 64bit System cause i didn't have one.

I tested it on :

Win32 PHP 4.3.x
Win32 PHP 5.0.0
Linux64 PHP 5.0.5
Linux32 PHP 5.0.5

and now it works on all servers.

Additional, i coded the get_url() function, which was used to connect Google, out and replaced it with some fsockopen action. Cause on some Servers functions like get_url() are disabled for security reasons.

My Question is, if someone could test my changes on a 64bit PHP5.1.2 Machine and have a look to my code before i will publish the Code Changes here. (PM with E-Mail is enough) Cause i know i am not the best coder in Bit Operations(long long time ago) and i don't want to publish buggy PHP code here.

@Moderators: If my English is to bad, feel free to edit the post. I think it will be easier for others to read.

[myli]

Hi. Im very interested in an Pagerank Check Script in PHP for 64Bit systems.

All i could find was functions or classes for 32bit Systems.

Would be great if anyone have an idea.

by the way.. the checksum wich get created is on

32bit system: 1104488743
64bit system: -2627036557665167874


thats why i need a 64bit version

[jacki]

I am looking for this algorithm but in c++. Can somebody help me?

[CGSoftLabs]

Originally Posted by AboutSledge

PHP Data Type Converting Bug.


<?php

//header("Content-Type: text/plain; charset=utf-8");

$n = -6288256054;
var_dump($n);

$a = intval($n);
var_dump($a);

$b = (int) $n;
var_dump($b);

settype($n,"int");
var_dump($n);

?>

============good==========
float(-6288256054)
int(-1993288758)
int(-1993288758)
int(-1993288758)


============bad=================

float(-6288256054)
int(-2147483648)
int(-2147483648)
int(-2147483648)

yes..this is the problem
I have a php script running on some server for getting pagerank;
After running fine for a while, few weeks ago it died; so I said..google changed the crc..but it wasn't like that; yesterday I started to read the forums in search of answers till I found this;

I looked deep inside the code to see what's really happens
First let's say that php installed on my server is PHP Version 4.3.10;
Well the problem lies inside mix($a,$b,$c) function; it's about Arithmetic Operators which can operate only with 32bit balues; so when we perform $a ^ (zeroFill($c,13)) we must be sure we have 32bit integers;
On 32bit OS's the conversion is done well from double (64bit numbers) to long (32 bit) before xoring; but on 64 you see what's happens especially with small negative numbers ..small than MIN_INT which is (-2147483648) aka 0x80000000;

so if we have a double (-6288256054) which is small than MIN_INT on 32OS will be truncated in a way..and on 64 it will return always MIN_INT; knowing this we can detect where it runs on 64 or 32 bit OS's ;

so quiqly I write a helper to fix our double integers;

PHP Code:

function toInt32(& $x){
    $z = hexdec(80000000);
    $y = (int)$x;
   // on 64bit OSs if $x is double, negative ,will return -$z in $y
   // which means 32th bit set (the sign bit)
    if($y==-$z&&$x<-$z){
         $y = (int)((-1)*$x);// this is the hack, make it positive before
         $y = (-1)*$y; // switch back the sign
         //echo "int hack <br>";
    }
    $x = $y;
} 


then apply this function right before xoring inside mix function...foreach line

PHP Code:

 $a -= $b; $a -= $c; toInt32($a); $a = (int)($a ^ (zeroFill($c,13)));

function mix($a,$b,$c) {
  $a -= $b; $a -= $c; toInt32($a); $a = (int)($a ^ (zeroFill($c,13)));
  $b -= $c; $b -= $a; toInt32($b); $b = (int)($b ^ ($a<<8));
  $c -= $a; $c -= $b; toInt32($c); $c = (int)($c ^ (zeroFill($b,13)));
  $a -= $b; $a -= $c; toInt32($a); $a = (int)($a ^ (zeroFill($c,12)));
  $b -= $c; $b -= $a; toInt32($b); $b = (int)($b ^ ($a<<16));
  $c -= $a; $c -= $b; toInt32($c); $c = (int)($c ^ (zeroFill($b,5)));
  $a -= $b; $a -= $c; toInt32($a); $a = (int)($a ^ (zeroFill($c,3)));
  $b -= $c; $b -= $a; toInt32($b); $b = (int)($b ^ ($a<<10));
  $c -= $a; $c -= $b; toInt32($c); $c = (int)($c ^ (zeroFill($b,15)));
  return array($a,$b,$c);
} 


with this, now I have my script running again...on "supposed" 64bit host machine..(coz I don't know if this is true )

greatz
CG
http://cgsoftlabs.ro

[truvahorse]

This php pagerank script works fine.

2 files and 1 directory in it.
index.php (calls pagerank.php)
pagerank.php (script code)
images (images directory)

Upload all files and run "index.php"

Now who can help me to convert "index.php" codes to html codes.
Becasue i want to call "pagerank.php" by a html code and and offer this little code to my visitors.

Regards,

[Jippi]

Your client does not have permission t<>o get URL /search?client=navclient-auto&ch=6-1392492760&features=Rank&q=info:http://www.guu.dk/ from t<>his server. (Client IP address: 82.103.133.192)

With that script

PHP 5.1.2 (cli) (built: Feb 15 2006 18:39:30)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v2.1.0, Copyright (c) 1998-2006 Zend Technologies
with eAccelerator v0.9.5-beta1, Copyright (c) 2004-2005 eAccelerator, by eAccelerator

[Jippi]

I applied CGSoftLabs 's toInt32 patch, and now it works like a charm

[myli]

may you post it please? i tried it but it didnt work. im running PHP Version 4.3.10
with apache 2.0 and suse 9.0 64bit.

if i use it on 32bit it works but when i try to use it on 64bit its says "forbidden". i.e for google.com it tries to open http://www.google.com/search?client=...www.google.com
on 64bit

[ontananza]

I've already looking for a google checksum algorithm, i've found one:

Written and contributed by
Alex Stapleton,
Andy Doctorow,
Tarakan,
Bill Zeller,
Vijay "Cyberax" Bhatter
traB

header("Content-Type: text/plain; charset=utf-8");
define('GOOGLE_MAGIC', 0xE6359A60);

etc...

I've tested on local enviroment on windows (PHP 5), it works :

but when i moved out to one linux server with php 4.3.9 it doesn't works

im not sure why is it...

someone could tell me why???

[Gagget]

Sorry, I have completly forgotten to post my code here. It´s tested by some people now and it works very fine.
To make it easier to update existing scripts, i commented the changes.

PHP Code:

/* 
        Written and contributed by 
        Alex Stapleton, 
        Andy Doctorow, 
        Tarakan, 
        Bill Zeller, 
        Vijay "Cyberax" Bhatter 
        traB 
    This code is released into the public domain 
*/ 
define('GOOGLE_MAGIC', 0x00000000E6359A60); // CHANGED (64Bit)

//unsigned shift right 
function zeroFill($a, $b) 
{   
        $z = 0x0000000080000000;  // CHANGED (64Bit)
        $a = $a & 0x00000000FFFFFFFF;  // ADDED (64Bit)
        
        if ($z & $a) 
        {     
            $a = ($a>>1); 
            $a &= (~$z); 
            $a |= 0x0000000040000000;  // CHANGED (64Bit)
            $a = ($a>>($b-1)); 
        } 
        else 
        {     $a = ($a>>$b);    
        } 
        return $a; 
} 
  
  
function mix($a,$b,$c) { 
    
  $a = $a & 0x00000000FFFFFFFF;  // ADDED (64Bit)
  $b = $b & 0x00000000FFFFFFFF;  // ADDED (64Bit)
  $c = $c & 0x00000000FFFFFFFF;  // ADDED (64Bit)
  
  $a -= $b; $a -= $c; $a ^= (zeroFill($c,13)); 
  $b -= $c; $b -= $a; $b ^= ($a<<8); 
  $c -= $a; $c -= $b; $c ^= (zeroFill($b,13)); 
  $a -= $b; $a -= $c; $a ^= (zeroFill($c,12)); 
  $b -= $c; $b -= $a; $b ^= ($a<<16); 
  $c -= $a; $c -= $b; $c ^= (zeroFill($b,5)); 
  $a -= $b; $a -= $c; $a ^= (zeroFill($c,3));   
  $b -= $c; $b -= $a; $b ^= ($a<<10); 
  $c -= $a; $c -= $b; $c ^= (zeroFill($b,15)); 

  return array($a,$b,$c); 
} 
 
 
function GoogleCH($url, $length=null, $init=GOOGLE_MAGIC) { 
    if(is_null($length)) { 
        $length = sizeof($url); 
    } 
    $a = $b = 0x000000009E3779B9;  // CHANGED (64Bit)
    
    $c = $init ; 
    $k = 0; 
    $len = $length; 
    while($len >= 12) { 
        $a += ($url[$k+0] +($url[$k+1]<<8) +($url[$k+2]<<16) +($url[$k+3]<<24)); 
        $b += ($url[$k+4] +($url[$k+5]<<8) +($url[$k+6]<<16) +($url[$k+7]<<24)); 
        $c += ($url[$k+8] +($url[$k+9]<<8) +($url[$k+10]<<16)+($url[$k+11]<<24)); 
        $mix = mix($a,$b,$c); 
        $a = $mix[0]; $b = $mix[1]; $c = $mix[2]; 
        $k += 12; 
        $len -= 12; 
    } 

    $c += $length; 
    switch($len)              /* all the case statements fall through */ 
    { 
        case 11: $c+=($url[$k+10]<<24); 
        case 10: $c+=($url[$k+9]<<16); 
        case 9 : $c+=($url[$k+8]<<8); 
          /* the first byte of c is reserved for the length */ 
        case 8 : $b+=($url[$k+7]<<24); 
        case 7 : $b+=($url[$k+6]<<16); 
        case 6 : $b+=($url[$k+5]<<8); 
        case 5 : $b+=($url[$k+4]); 
        case 4 : $a+=($url[$k+3]<<24); 
        case 3 : $a+=($url[$k+2]<<16); 
        case 2 : $a+=($url[$k+1]<<8); 
        case 1 : $a+=($url[$k+0]); 
         /* case 0: nothing left to add */ 
    } 
    $mix = mix($a,$b,$c); 
    /*-------------------------------------------- report the result */ 
    return $mix[2]; 
} 

//converts a string into an array of integers containing the numeric value of the char 
function strord($string) { 
    for($i=0;$i<strlen($string);$i++) { 
        $result[$i] = ord($string{$i}); 
    } 
    return $result; 
} 

// converts an array of 32 bit integers into an array with 8 bit values. Equivalent to (BYTE *)arr32 
function c32to8bit($arr32) { 
    for($i=0;$i<count($arr32);$i++) { 
        for ($bitOrder=$i*4;$bitOrder<=$i*4+3;$bitOrder++) { 
            $arr8[$bitOrder]=$arr32[$i]&255; 
            $arr32[$i]=zeroFill($arr32[$i], 8); 
        }     
    } 
    return $arr8; 
} 


function GoogleCHNew($ch){
    $ch=sprintf("%u", $ch); 
    $ch = ((($ch/7) << 2) | (((int)fmod($ch,13))&7)); 
    $prbuf = array(); 
    $prbuf[0] = $ch; 
    for($i = 1; $i < 20; $i++) { 
      $prbuf[$i] = $prbuf[$i-1]-9; 
    } 
    $ch = GoogleCH(c32to8bit($prbuf), 80); 
    return sprintf("%u", $ch);
}


function get_page_rank($url){
    $url = preg_replace('/\?.*$/','?',$url);
    $reqgr = "info:".$url;
    $reqgre = "info:".urlencode($url);
    $gch = GoogleCH(strord($reqgr));
    
    $gch = "6".GoogleCHNew($gch);
    

  
    $patern = '/^http:/';
    $patern2 = '/^http:\/\/.*google\..*\/(search|images|groups|news).*/';
    $patern3 = '/^http:\/\/localhost.*/';
    $patern4 = '/^http:\/\/(127\.|10\.|172\.16|192\.168).*/'; //local ip
    if(!preg_match($patern, $url) || preg_match($patern2, $url) ||
       preg_match($patern3, $url) || preg_match($patern4, $url)){
           return -1;
    }else{
        // BEGIN CHANGES (fsockopen to request PR) 
        
        $fsock = fsockopen('toolbarqueries.google.com', 80, $errno, $errstr);
        if ( !$fsock ){  
            return -1;
        }
        $base_get = "/search?client=navclient-auto&ch=".$gch."&ie=UTF-8&oe=UTF-8&features=Rank:FVN&q=".$reqgre;
        fputs($fsock, "GET $base_get HTTP/1.1\r\n");
        fputs($fsock, "HOST: toolbarqueries.google.com\r\n");
        fputs($fsock, "User-Agent: Mozilla/4.0 (compatible; GoogleToolbar 2.0.114-big; Windows XP 5.1)\r\n");
        fputs($fsock, "Connection: close\r\n\r\n");
        while(!feof($fsock)){ 
            $res['content'] .= fread($fsock, 1024);
        }
        fclose($fsock);

        // END CHANGES (fsockopen to request PR)
    
        if(preg_match('/Rank_.*?:.*?:(\d+)/i', $res['content'], $m)){    
            return $m[1];
        }else{    
            return -1;
        }      
    }
} 
?>