An external fix--[Kindle, periodical] Remember last article selected

nickredding · 07-14-2011, 11:29 PM

This problem (see this thread) has proven impossible to solve (at least for me) via reverse-engineering the MOBI format variant being used by Amazon. Therefore, because I consider the problem a serious impairment of the Kindle 3 user interface, I have developed a software engineering solution which works quite well. I'm sharing it here for the benefit of others who want a solution to this (for me) vexing problem!

The solution consists of using the Calibre command line interface to create a standard OEB version of a downloaded periodical and then processing that with Amazon's Kindlegen to produce the Kindle-compatible MOBI file. I have automated this process with a Windows batch file. If you want to schedule downloads you can use the Windows scheduler to run the batch file on whatever schedule you wish.

The batch file edits the recipe you specify to add some code to make the Calibre OEB output work properly with Kindlegen. You don't have to worry about what those changes are, and no changes are made to your original copy of your custom recipe (if you're using one) or the built-in recipe (if you're using one of those). The changes are made and stored in a temporary recipe file every time you run the batch file.

The batch file will optionally add the generated files to the Calibre library and/or email them to your Kindle (or anywhere else you choose).

The disadvantages of this solution are (a) you have to download and install Python and Kindlegen on your PC--but they are both free and easy to install; (b) it runs outside of Calibre; (c) you have to reference built-in recipes by their files names in the Calibre release directory; (d) it's a PC-only solution--sorry, Apple people; (e) the masthead feature doesn't work--a limitation of Kindlegen; (f) I don't have time to provide hand-holding support for this.

The advantages of this solution are (a) it works--problem solved; (b) it uses whatever release version of Calibre you're running, including the built-in recipes, so you don't have to do anything to stay in sync with Calibre.

There are complete instructions in the attached zipped folder for setting up and using this solution. If you find bugs please let me know, and I'll post updates in this thread as necessary.

Starson17 · 07-15-2011, 09:12 AM

You may want to post this in the sticky thread for re-usable code.
https://www.mobileread.com/forums/showthread.php?t=70263

tylau0 · 07-15-2011, 07:46 PM

Nickredding, it's interesting to see we are working on the same thing! While you are working on the windows version, I am working on the Linux version. Attached please find my program. Cheer!

veezh · 07-16-2011, 05:11 AM

Thank you, nick, for what looks like a great solution. I've tried it now a couple of times with two built-in recipes (le_monde and economist), but I get the following error each time in the log:

Python function terminated unexpectedly
global name 'urlparse' is not defined (Error Code: 1)

Here's what I'm getting at the command prompt in case it helps:

Spoiler:

Do I need to make changes to the recipes to get them to work?

Many thanks in advance.

nickredding · 07-16-2011, 04:54 PM

Quote:

Originally Posted by veezh

Thank you, nick, for what looks like a great solution. I've tried it now a couple of times with two built-in recipes (le_monde and economist), but I get the following error each time in the log:

Python function terminated unexpectedly
global name 'urlparse' is not defined (Error Code: 1)

veezh -- Thanks for the heads-up. There was an unresolved dependency (I'm not sure why I didn't run into that myself) but the fix is in the attached zip file (note that news.bat has also been updated to process several recipes sequentially). You should replace the files in Calibre-Kindle\Recipes\Resources

veezh · 07-17-2011, 05:16 AM

Thanks for the new version, nick. I've just given it a go with both the recipes I was trying before. Another error is being generated now:

Python function terminated unexpectedly
[Errno 2] No such file or directory: 'C:\\Documents and Settings\\veezh\\Calibre-Kindle\\Recipes\\Temp\\economist.ncx' (Error Code: 1)

Here's what I'm seeing at the command prompt:

Spoiler:

Please let me know if you need any more info. Really looking forward to getting this up and running!

nickredding · 07-17-2011, 11:58 AM

Quote:

Originally Posted by veezh

Thanks for the new version, nick. I've just given it a go with both the recipes I was trying before. Another error is being generated now:

Python function terminated unexpectedly
[Errno 2] No such file or directory: 'C:\\Documents and Settings\\veezh\\Calibre-Kindle\\Recipes\\Temp\\economist.ncx' (Error Code: 1)

veezh -- I can't reproduce that one. I just ran "NEWS economist" to generate the magazine using the builtin-in recipe and it ran properly. However, the directory the recipe is trying to use looks strange. It should look like 'C:\users\veezh\Calibre-Kindle\ ...' so I don't know how it ended up with 'C:\Documents and Settings\veezh\Calibre-Kindle\ ...''. The error arose because the latter path doesn't exist.

I'll look into this, so check back later and I'll have it sorted out.

veezh · 07-17-2011, 12:20 PM

Quote:

Originally Posted by nickredding

However, the directory the recipe is trying to use looks strange. It should look like 'C:\users\veezh\Calibre-Kindle\ ...' so I don't know how it ended up with 'C:\Documents and Settings\veezh\Calibre-Kindle\ ...''.

I thought that was weird, too, but just assumed it had something to do with the temp directory calibre uses when generating publications (?). Will check back later -- thanks again!

tylau0 · 07-17-2011, 01:59 PM

For a user named xxx, Windows XP uses "C:\Documents and Settings\xxx" as the default home folder, while Windows Vista/7 uses "C:\Users\xxx". You may need to change those "C:\Users" with "C:\Documents and Settings\" in the script if you use Windows XP instead of Windows Vista/7.

nickredding · 07-17-2011, 04:08 PM

Quote:

Originally Posted by tylau0

For a user named xxx, Windows XP uses "C:\Documents and Settings\xxx" as the default home folder, while Windows Vista/7 uses "C:\Users\xxx". You may need to change those "C:\Users" with "C:\Documents and Settings\" in the script if you use Windows XP instead of Windows Vista/7.

Yes, I failed to account for Windows XP. I have attached an updated zip file with modified files to account for this. You should replace and update ALL the files if you want to use them.

However, at the moment there is a problem with Windows XP (I tested the files on an old XP computer I have but rarely use): it seems ebook-convert corrupts the command environment on XP and this causes the news batch file to termimate. I verified this by manually running ebook-convert: it completes the download and conversion, but it corrupts the command.exe environment in such a way that batch files exit without doing anything and you are unable to run any other batch files. To run a batch file again you have to close the window and open a new command prompt window.

Now, I'm assuming this problem is not unique to my old XP computer, but I'd obviously be interested in veezh or anyone else trying this and seeing if it happens on your XP system. The symptom is

NEWS recipename

terminates in your top-level user directory after displaying

Downloading periodical recipename ...

If you then type "NEWS" (or the name of any other batch file) the command line window simply does nothing (seems to ignore the command).

I will post a message in the main Calibre forum to ask Kovid about this. Until he can issue a fix I don't think XP users will have any joy with this solution. Of course, you could always jump ahead 10 years and upgrade to a Windows 7 computer!

nickredding · 07-17-2011, 05:46 PM

Thanks to a quick response from dwanthny in the main forum I have now attached a solution that works under XP as well as Windows 7.

oneillpt · 07-17-2011, 07:14 PM

I've made a change to one line of news.bat:

calibredb add %fname%.mobi --duplicates >%fname%-add-log.txt
(adding --duplicates)

as without this change my Calibre library is not updated when a new file is generated, as shown in the log fragment here:

Code:

reply: retcode (221); Msg: 2.3.0 Bye received. Goodbye.
Adding ITimes.mobi to Calibre library ...
The following books were not added as they already exist in the database (see --duplicates option):
        The Irish Times (2) *:
                  C:\Users\Peter\Calibre-Kindle\Recipes\Temp\ITimes.oeb\ITimes.mobi

Recipe processing complete

A further hint which might be useful for users in case of e-mail problems: my first attempt to have the generated file e-mailed failed because I had supplied my "user friendly alias" e-mail address in news-set-vars.bat, but authentication with my mail server through an ISP failed as this should have been the less friendly account name. My second attempt also failed - I had forgotten that the "Approved Personal Document E-mail List" for my Kindle account at Amazon only contained the "user friendly" version, so I needed to add the version using the less friendly account name. Obvious of course in hindsight, but it took me a couple of minutes to realise what the problem was.

nickredding · 07-17-2011, 07:44 PM

Quote:

Originally Posted by oneillpt

I've made a change to one line of news.bat:

calibredb add %fname%.mobi --duplicates >%fname%-add-log.txt (adding --duplicates)

Thanks for that suggestion--I have updated the news.bat file.

I'm not sure what you mean by "user friendly alias" for e-mail addresses. Assuming you are using a server (such as gmail) the parameters in news-set-vars.bat should be set exactly as they are in the Calibre GUI. If you can clarify this for me I'll update the Instructions file.

nickredding · 07-17-2011, 07:55 PM

Note to people using this solution: you've probably noticed that the formatting of the output (as seen on your Kindle) from news.bat is slightly different from the output from the Calibre GUI. That is because the MOBI file is being generated by Kindlegen from the OEB form of the periodical. Kindlegen is more primitive than the Calibre MOBI output code and some of the CSS directives that are properly recognized by Calibre are flubbed by Kindlegen. I'm working on modifying most of my frequently-used news recipes to account for Kindlegen's limitations and when I have enough information to explain how to do this reliably I'll post it in this forum for anyone who wants to do the same for their favourite recipes.

oneillpt · 07-17-2011, 08:10 PM

Quote:

Originally Posted by tylau0

Nickredding, it's interesting to see we are working on the same thing! While you are working on the windows version, I am working on the Linux version. Attached please find my program. Cheer!

Just a note to say that when your shell script is translated to a .bat file your Linux version runs under Windows 7 too.

07-14-2011, 11:29 PM	#1
nickredding onlinenewsreader.net Posts: 324 Karma: 10143 Join Date: Dec 2009 Location: Phoenix, AZ & Victoria, BC Device: Kindle 3, Kindle Fire, IPad3, iPhone4, Playbook, HTC Inspire	An external fix--[Kindle, periodical] Remember last article selected This problem (see this thread) has proven impossible to solve (at least for me) via reverse-engineering the MOBI format variant being used by Amazon. Therefore, because I consider the problem a serious impairment of the Kindle 3 user interface, I have developed a software engineering solution which works quite well. I'm sharing it here for the benefit of others who want a solution to this (for me) vexing problem! The solution consists of using the Calibre command line interface to create a standard OEB version of a downloaded periodical and then processing that with Amazon's Kindlegen to produce the Kindle-compatible MOBI file. I have automated this process with a Windows batch file. If you want to schedule downloads you can use the Windows scheduler to run the batch file on whatever schedule you wish. The batch file edits the recipe you specify to add some code to make the Calibre OEB output work properly with Kindlegen. You don't have to worry about what those changes are, and no changes are made to your original copy of your custom recipe (if you're using one) or the built-in recipe (if you're using one of those). The changes are made and stored in a temporary recipe file every time you run the batch file. The batch file will optionally add the generated files to the Calibre library and/or email them to your Kindle (or anywhere else you choose). The disadvantages of this solution are (a) you have to download and install Python and Kindlegen on your PC--but they are both free and easy to install; (b) it runs outside of Calibre; (c) you have to reference built-in recipes by their files names in the Calibre release directory; (d) it's a PC-only solution--sorry, Apple people; (e) the masthead feature doesn't work--a limitation of Kindlegen; (f) I don't have time to provide hand-holding support for this. The advantages of this solution are (a) it works--problem solved; (b) it uses whatever release version of Calibre you're running, including the built-in recipes, so you don't have to do anything to stay in sync with Calibre. There are complete instructions in the attached zipped folder for setting up and using this solution. If you find bugs please let me know, and I'll post updates in this thread as necessary. Last edited by nickredding; 07-17-2011 at 01:59 AM.

07-16-2011, 05:11 AM	#4
veezh plus ça change Posts: 101 Karma: 32134 Join Date: Dec 2009 Location: France Device: Kindle PW2, Voyage	Thank you, nick, for what looks like a great solution. I've tried it now a couple of times with two built-in recipes (le_monde and economist), but I get the following error each time in the log: Python function terminated unexpectedly global name 'urlparse' is not defined (Error Code: 1) Here's what I'm getting at the command prompt in case it helps: Spoiler: C:\users\veezh>news le_monde -email -add -user userid password Updating recipe ... Downloading periodical ... Traceback (most recent call last): File "site.py", line 132, in main File "site.py", line 109, in run_entry_point File "site-packages\calibre\ebooks\conversion\cli.py", line 283, in main File "site-packages\calibre\ebooks\conversion\plumber.py", line 930, in run File "site-packages\calibre\customize\conversion.py", line 204, in __call__ File "site-packages\calibre\web\feeds\input.py", line 105, in convert File "site-packages\calibre\web\feeds\news.py", line 735, in download File "site-packages\calibre\web\feeds\news.py", line 977, in build_index File "c:\docume~1\veezh\locals~1\temp\calibre_0.8.10_tm p_waq937\calibre_0.8 .10_cff8kc_recipes\recipe0.py", line 79, in feed2index bn = urlparse.urlsplit(feed.image_url).path NameError: global name 'urlparse' is not defined C:\users\veezh\Calibre-Kindle\Recipes\Temp> Do I need to make changes to the recipes to get them to work? Many thanks in advance.

07-17-2011, 05:16 AM	#6
veezh plus ça change Posts: 101 Karma: 32134 Join Date: Dec 2009 Location: France Device: Kindle PW2, Voyage	Thanks for the new version, nick. I've just given it a go with both the recipes I was trying before. Another error is being generated now: Python function terminated unexpectedly [Errno 2] No such file or directory: 'C:\\Documents and Settings\\veezh\\Calibre-Kindle\\Recipes\\Temp\\economist.ncx' (Error Code: 1) Here's what I'm seeing at the command prompt: Spoiler: C:\users\veezh>news economist -email -add -user userid password Recipe economist numfiles=1 email=yes add=yes user=yes userid=userid password=password pause= verbose= options= Updating recipe economist ... Downloading periodical economist ... Traceback (most recent call last): File "site.py", line 132, in main File "site.py", line 109, in run_entry_point File "site-packages\calibre\ebooks\conversion\cli.py", line 283, in main File "site-packages\calibre\ebooks\conversion\plumber.py", line 930, in run File "site-packages\calibre\customize\conversion.py", line 204, in __call__ File "site-packages\calibre\web\feeds\input.py", line 105, in convert File "site-packages\calibre\web\feeds\news.py", line 735, in download File "site-packages\calibre\web\feeds\news.py", line 981, in build_index File "c:\docume~1\veezh\locals~1\temp\calibre_0.8.10_tm p_9gbwrf\calibre_0.8 .10_bujkiy_recipes\recipe0.py", line 60, in create_opf alt_ncx_stream = open(alt_ncx_path,'wb') IOError: [Errno 2] No such file or directory: 'C:\\Documents and Settings\\veezh\\Calibre-Kindle\\Recipes\\Temp\\economist.ncx' C:\users\veezh> Please let me know if you need any more info. Really looking forward to getting this up and running!

07-17-2011, 05:46 PM	#11
nickredding onlinenewsreader.net Posts: 324 Karma: 10143 Join Date: Dec 2009 Location: Phoenix, AZ & Victoria, BC Device: Kindle 3, Kindle Fire, IPad3, iPhone4, Playbook, HTC Inspire	Thanks to a quick response from dwanthny in the main forum I have now attached a solution that works under XP as well as Windows 7. Last edited by nickredding; 07-17-2011 at 07:47 PM.

07-17-2011, 07:14 PM	#12
oneillpt Connoisseur Posts: 62 Karma: 46 Join Date: Feb 2011 Device: Kindle 3 (cracked screen!); PW1; Oasis	A small problem solved I've made a change to one line of news.bat: calibredb add %fname%.mobi --duplicates >%fname%-add-log.txt (adding --duplicates) as without this change my Calibre library is not updated when a new file is generated, as shown in the log fragment here: Code: reply: retcode (221); Msg: 2.3.0 Bye received. Goodbye. Adding ITimes.mobi to Calibre library ... The following books were not added as they already exist in the database (see --duplicates option): The Irish Times (2) *: C:\Users\Peter\Calibre-Kindle\Recipes\Temp\ITimes.oeb\ITimes.mobi Recipe processing complete A further hint which might be useful for users in case of e-mail problems: my first attempt to have the generated file e-mailed failed because I had supplied my "user friendly alias" e-mail address in news-set-vars.bat, but authentication with my mail server through an ISP failed as this should have been the less friendly account name. My second attempt also failed - I had forgotten that the "Approved Personal Document E-mail List" for my Kindle account at Amazon only contained the "user friendly" version, so I needed to add the version using the less friendly account name. Obvious of course in hindsight, but it took me a couple of minutes to realise what the problem was.

07-15-2011, 09:12 AM	#2
Starson17 Wizard Posts: 4,004 Karma: 177841 Join Date: Dec 2009 Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T	You may want to post this in the sticky thread for re-usable code. https://www.mobileread.com/forums/showthread.php?t=70263

07-17-2011, 01:59 PM	#9
tylau0 Connoisseur Posts: 82 Karma: 10 Join Date: Oct 2010 Device: Kindle	For a user named xxx, Windows XP uses "C:\Documents and Settings\xxx" as the default home folder, while Windows Vista/7 uses "C:\Users\xxx". You may need to change those "C:\Users" with "C:\Documents and Settings\" in the script if you use Windows XP instead of Windows Vista/7.

07-17-2011, 07:55 PM	#14
nickredding onlinenewsreader.net Posts: 324 Karma: 10143 Join Date: Dec 2009 Location: Phoenix, AZ & Victoria, BC Device: Kindle 3, Kindle Fire, IPad3, iPhone4, Playbook, HTC Inspire	Note to people using this solution: you've probably noticed that the formatting of the output (as seen on your Kindle) from news.bat is slightly different from the output from the Calibre GUI. That is because the MOBI file is being generated by Kindlegen from the OEB form of the periodical. Kindlegen is more primitive than the Calibre MOBI output code and some of the CSS directives that are properly recognized by Calibre are flubbed by Kindlegen. I'm working on modifying most of my frequently-used news recipes to account for Kindlegen's limitations and when I have enough information to explain how to do this reliably I'll post it in this forum for anyone who wants to do the same for their favourite recipes. Last edited by nickredding; 07-17-2011 at 08:01 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Kindle, periodical] Remember last article selected	tylau0	Recipes	29	07-22-2011 02:23 PM
Free (Kindle) Remember Why You Play	arcadata	Deals and Resources (No Self-Promotion or Affiliate Links)	0	10-04-2010 08:17 PM
Gizmodo article: How To Fix Today's Ebook readers	ekaser	News	76	04-30-2010 05:10 AM
Kindle Document as Periodical	Gideon	Amazon Kindle	2	04-23-2008 06:50 PM
Remember Kindle ?	yvanleterrible	Amazon Kindle	14	09-12-2007 09:05 AM

Advert

Advert