Auto loading meta data from Amazon

[m0ngr31]

Hey guys, if you've been following the Sideloading and Whispersync thread, you've seen that there have been some cool new developments about getting your books (not purchased from Amazon) to sync across different devices.

It turns out that Amazon will sync your books across multiple devices if they appear to have a legit ASIN number. But if you have an Android or iOS device, it would sometimes not download the correct book cover because it couldn't associate that ASIN number with a Kindle book. So to solve that problem, I wrote a couple of scripts (one for Windows and one for *nix). They will automatically download the correct ASIN number from Amazon. If it cannot find a match, the script will just give you a random ASIN number so it will still sync between your devices.

Let me walk you through doing this:

-Backup all your .mobi files before trying this!
-Get an Amazon Web Services account. It takes about all of 2 seconds. You'll need the API and Secret keys

Windows:
-Download and install Python
-Download and install SetupTools
-Install python-amazon-product-api
-Make sure C:\python27 and c:\python27\scripts are in your PATH Environment
-If you can't do the "easy_install" download and install this
-Put all your .mobi files in a single folder
-Download MobiPerl and extract it to the same folder your .mobi files are in
-Save this file as a .bat file in that same folder:

@echo off & setLocal EnableDELAYedExpansion

del temp*.txt>null 2>&1
set "asin1=B00"
set /a count=0

for /f %%a IN ('dir /b *.mobi') do (set /a count=count+1)
echo Number of files: %count%

set /a diff=((9999-%count%) - 1000) + 1
set /a divisor = 32767 / %diff%
set /a diff2=(999 - 100) + 1
set /a divisor2 = 32767 / %diff2%

:Random1
set /a asin2=%random% / %divisor%
if /i %asin2% GTR 9999 goto Random1
if /i %asin2% LSS 1000 goto Random1

:Random2
set /a asin3=%random% / %divisor2%
if /i %asin3% GTR 999 goto Random2
if /i %asin3% LSS 100 goto Random2
set /a temp1=!asin2!

for /f "delims=  " %%b IN ('dir /b *.mobi') do (

C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --outfile "%%b" --delexthtype 113 "%%b">null 2>&1
C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --outfile "%%b" --delexthtype 501 "%%b">null 2>&1

C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe "%%b" > temp1.txt
findstr /i "longtitle" temp1.txt > temp2.txt
set /p title1=<temp2.txt
set title1=!title1:~11!

find /i "item: 100" temp1.txt > temp3.txt

for /f "tokens=* delims= " %%a in (temp3.txt) do (
set var=%%a
)

set author1=!var!
set author1=!author1:~33!

python asin_search.py "!title1!" "!author1!" >temp4.txt

set /p temp2=<temp4.txt
set temp2=!temp2:~0!

set filename=%%b
call :movement %%b
)

:movement

if "!tempfile!"=="!filename!" goto :end

find /c /i "b0" "temp4.txt">null 2>&1
if !ERRORLEVEL! EQU 0 goto :real

:fake
C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --outfile "!filename!" --exthtype 113 --exthdata "!asin1!!asin3!!temp1!" "!filename!">null 2>&1
echo Gave a fake ASIN number to: !title1!. It's new number is !asin1!!asin3!!temp1!
set /a temp1=!temp1!+1
goto final

:real
C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --outfile "!filename!" --exthtype 113 --exthdata "!temp2!" "!filename!">null 2>&1
echo Gave a real ASIN number to: !title1!. It's new number is !temp2!

:final
C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --outfile "!filename!" --addexthtype 501 --exthdata "EBOK" "!filename!">null 2>&1
set tempfile=!filename!

:end
del temp*.txt>null 2>&1
del null*

-Replace "C:\Users\m0ngr31\desktop\kindle\" with your working directory
-Download this python script and save it as asin_search.py in your working directory

import sys
from amazonproduct import API

AWS_KEY = 'A******************A'
SECRET_KEY = 'h*************************************7'
api = API(AWS_KEY, SECRET_KEY, "us")
node = api.item_search('KindleStore', Title=sys.argv[1], Author=sys.argv[2])

for book in node.Items.Item:
	print '%s' % (book.ASIN)

-Replace with your own API and Secret keys
-Run the .bat file and wait for it to finish.
-Copy the updated files to your kindle, android, iOS, etc.
-See if it worked!

*nix:
-Make sure mobiperl is installed correctly (MAC) (Linux)
-Install python-amazon-product-api
-Put all your .mobi files in a single folder
-Save the above python script as asin_search.py in your working directory
-Replace with your own API and Secret keys
-Save this bash script as a .sh file in your working directory

#!/bin/bash

count1=0
asin1="B00"

FILES="*.mobi"

for g in $FILES
do
	if [ "$g" ]
	then
		count1=$[$count1+1]
		continue
	fi
done
echo "There is $count1 files"

rand1=$[1000-$count1]

while [ $rand1 -lt 1000 ]
do
	rand1=$[ (( $RANDOM % ( $[ 9999 - 1000 ] + 1 ) ) + 1000) - $count1 ]
done

rand2=$[ ( $RANDOM % ( $[ 999 - 100 ] + 1 ) ) + 100 ]

for f in $FILES
do
	if [ "$f" ]
	then
		mobi2mobi --outfile "$f" --delexthtype 113 "$f"> /dev/null 2>&1
		mobi2mobi --outfile "$f" --delexthtype 501 "$f"> /dev/null 2>&1
		
		author=$(mobi2mobi "$f" | grep Author | cut -d- -f4)
		title=$(mobi2mobi "$f" | grep LONGTITLE | cut -c11-100)
		
		python asin_search.py "$title" "$author">asin.txt
		search1=`head -n 1 asin.txt`

		if [ "$search1" ]
		then
			mobi2mobi --outfile "$f" --exthtype 113 --exthdata $search1 "$f"> /dev/null 2>&1
			echo "Gave a real ASIN number to: $title. It's new number is $search1"
		else
			mobi2mobi --outfile "$f" --exthtype 113 --exthdata $asin1$rand2$rand1 "$f"> /dev/null 2>&1
			echo "Gave a fake ASIN number to: $title. It's new number is $asin1$rand2$rand1"
			rand1=$[$rand1+1]
		fi
		
		mobi2mobi --outfile "$f" --addexthtype 501 --exthdata "EBOK" "$f"> /dev/null 2>&1
		rm asin.txt
		continue
	fi
done

-Make sure the .sh file is executable (chmod +x amazon.sh)
-Copy the updated files to your kindle, android, iOS, etc.
-See if it worked!

If anyone gets some use out of this please let me know! My next project will be to get the covert art of the random ASIN numbers to work correctly on android, iOS, ect.

You guys are the best

[daffy4u]

Thanks for your continued work on this. I've added a link back to this tread to the wiki.

[m0ngr31]

I just hope someone can get some use out of it

[m0ngr31]

I've made an update to the script. This will make it so it saves the cover art of the book as cover_B00*******.png (like Kerenon suggested it needed to be for android) if it cannot find the correct ASIN from Amazon. This way you won't have to live with garbage cover art on your mobile devices.

Just put

mobi2mobi --savecover "cover_$asin1$rand2$rand1.png" "$f"> /dev/null 2>&1

between

mobi2mobi --outfile "$f" --exthtype 113 --exthdata $asin1$rand2$rand1 "$f"> /dev/null 2>&1
echo "Gave a fake ASIN number to: $title. It's new number is $asin1$rand2$rand1"

If I get ambitious I'll figure out how to make this a calibre plugin.


--EDIT--

For the .bat file:
Put

C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --savecover "cover_!asin1!!asin3!!temp1!.png" "!filename!">null 2>&1

between

C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --outfile "!filename!" --exthtype 113 --exthdata "!asin1!!asin3!!temp1!" "!filename!">null 2>&1
echo Gave a fake ASIN number to: !title1!. It's new number is !asin1!!asin3!!temp1!

I'm guessing that should work, but I didn't test it because I didn't want to reboot. If anyone runs into problems let me know.

[kalex]

Question for you. What does it search by? I just tried one of my mobi files and I found the book in the kindle store on the website but your batch file didn't find it and gave it a fake ASIN. I'm trying to figure out what search criteria it uses so i can fine tune it in the metadata.

Thanks for great work

Alex

[kalex]

Hmm in addition to that it doesn't seem to modify the data inside the mobi file. I loaded the file back into mobi2mobigui and EXTH 113 and 501 are both not there. So I may need a bit of troubleshooting assistance as my python scripting skills suck

[DiapDealer]

Question for you. What does it search by? I just tried one of my mobi files and I found the book in the kindle store on the website but your batch file didn't find it and gave it a fake ASIN.

It looks like it's using the LONG TITLE which comes at the very end of the PDB file format's Record0. A possible solution would be to try both the LONG TITLE and the contents of EXTH record 503 for a possible ASIN match. It's never going to be perfect as the titles can vary wildly anyway.

@m0ngr31
Is it possible to expand the python api search to include non-kindle store content? It might have better luck finding a match from it's pbook database.

[kalex]

Book i was working on exists in Amazon kindle store and has same Long Title as the amazon's book. Weird
Also for some reason its running the fake title twice and assigns 2 different fake ASINs
book in question is: Moscow Sting: A Novel

and here is the output from the batch file:

Number of files: 1
Traceback (most recent call last):
  File "asin_search.py", line 7, in <module>
    node = api.item_search('KindleStore', Title=sys.argv[1], Author=sys.argv[2])

  File "build\bdist.win32\egg\amazonproduct.py", line 489, in item_search
  File "build\bdist.win32\egg\amazonproduct.py", line 405, in _parse
amazonproduct.NoExactMatchesFound
Gave a fake ASIN number to: Moscow Sting: A Novel. It's new number is B004202788

Gave a fake ASIN number to: Moscow Sting: A Novel. It's new number is B004202789

[m0ngr31]

Originally Posted by kalex

Question for you. What does it search by? I just tried one of my mobi files and I found the book in the kindle store on the website but your batch file didn't find it and gave it a fake ASIN. I'm trying to figure out what search criteria it uses so i can fine tune it in the metadata.

Thanks for great work

Alex

It searches by Longtitle + Author name. It's not a perfect science, but if you have them tagged correctly it should work most of the time. I ran it against 60 books and found the ones it was supposed to and didn't find the ones that amazon doesn't sell. Can you PM me the file your having problems with? I'll run it on my end and see if it works. Also Windows or *nix?

Also, I'm not sure why it would give 2 fake ASIN numbers. Probably crappy scripting on my part

[m0ngr31]

Originally Posted by DiapDealer

@m0ngr31
Is it possible to expand the python api search to include non-kindle store content? It might have better luck finding a match from it's pbook database.

I'll work on it

[kalex]

m0ngr31 sent you a PM. I'm in winblows

[m0ngr31]

I fixed the windows script. There were some serious logic errors I didn't catch because I didn't test enough haha. Just use the script in the first post.

[kalex]

Perfecto

[Feanor]

It's working great on Windows here, thank you so much, just one thing - I've added in the bit to create the cover_*.png, but it's not doing creating anything - any ideas why?

[m0ngr31]

If you put the save cover part in the proper place, it will only save a cover if it can't find the ASIN number. Is it for one that can be found. If it has a valid ASIN number, your Android/iOS device should download it automatically.

Edit:

I just thought of this. I did change the script a little. The modified line that you should put in is this.

C:\Users\m0ngr31\desktop\kindle\mobi2mobi.exe --savecover "cover_!asin1!!asin3!!temp1!.png" "!filename!">null 2>&1

Let me know if that works I've edited the post above to reflect the changes.