Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 10-28-2009, 11:07 AM   #1
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Question Automatic news feeds uploading, How does it work?

Hello Kovid,

Atleast two people, , are interested in how you detect/keep track of what needs to be uploaded to a connected reader.
For example:
If I download some newsfeeds and have a look in the database then I can't find anything different from regular books that are in there. Still when i connect my reader those feedbooks will be uploaded and deleted, if that preference is set.
I had a look at the source but this is beyond my python skills to find out myself.

The reason I'm curious is that it would be nice to be able to add an epub of a newspaper through calibre-db.exe and have that automatically uploaded when the reader connects.

Could you explain or give a pointer to the relevant part of the calibre source.

Thanks,

Joop
JvdW is offline   Reply With Quote
Old 10-28-2009, 11:38 AM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,778
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Uses the tag News (or the translation of it in the local language)
kovidgoyal is online now   Reply With Quote
Old 10-28-2009, 05:22 PM   #3
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Quote:
Originally Posted by kovidgoyal View Post
Uses the tag News (or the translation of it in the local language)
I'm pretty sure I tried that because that seemed logical todo. I'll give it another try tomorrow morning. (GMT+1)

Thanks,

Joop
JvdW is offline   Reply With Quote
Old 10-28-2009, 05:23 PM   #4
veysey
Enthusiast
veysey doesn't litterveysey doesn't litter
 
Posts: 24
Karma: 144
Join Date: Aug 2009
Location: Washington DC
Device: Sony ?
Can this be combined with other tags?

I have tried to do this too, using the tag news, but in conjunction with other tags as well. e.g. "News, Humor". Did not seem to automatically upload. Will this only work for "pure" news feeds?

Cheers,

John
veysey is offline   Reply With Quote
Old 10-28-2009, 05:28 PM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,778
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Actually, I'm sorry, what happens is that when a news download completes, the id is recorded in a config file which is then used to sync

Assuming you can find out the id of the freshly added news (which should be easy to do using calibredb, the following command will mark it to be synced

Code:
calibre-debug -c "from calibre.gui2 import dynamic; ids = dynamic.get('news_to_be_synced', set([])); ids.add(35); dynamic.set('news_to_be_synced', ids)"
replace 35 with the id of the freshly added news
kovidgoyal is online now   Reply With Quote
Old 10-29-2009, 11:04 AM   #6
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Quote:
Originally Posted by kovidgoyal View Post
Actually, I'm sorry, what happens is that when a news download completes, the id is recorded in a config file which is then used to sync

Assuming you can find out the id of the freshly added news (which should be easy to do using calibredb, the following command will mark it to be synced

Code:
calibre-debug -c "from calibre.gui2 import dynamic; ids = dynamic.get('news_to_be_synced', set([])); ids.add(35); dynamic.set('news_to_be_synced', ids)"
replace 35 with the id of the freshly added news
Yes, that works provided that Calibre itself is not running when you run this commandline.

Ofcourse Calibre knows its a newsfeed because its tagged News, what I'm wondering about is if it might be possible to drop that requirement?

Regards,

Joop
JvdW is offline   Reply With Quote
Old 10-29-2009, 11:08 AM   #7
ankit
Member
ankit began at the beginning.
 
Posts: 14
Karma: 10
Join Date: Oct 2009
Location: Cambridge, MA
Device: Sony PRS-300
A somewhat related question: is it possible for Calibre to delete the old version of the corresponding news from the device? I would like to keep only the current news version on the device. Checking to see if something like this is already hiding in the options somewhere.
ankit is offline   Reply With Quote
Old 10-29-2009, 11:54 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,778
Karma: 4998511
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
syncing is on the TODO list. When it is implemented, all these things become automatic.
kovidgoyal is online now   Reply With Quote
Old 10-29-2009, 02:04 PM   #9
quisvir
PRS+ contributor
quisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animals
 
quisvir's Avatar
 
Posts: 237
Karma: 6875
Join Date: Feb 2009
Location: Netherlands
Device: Sony 650 & T1
Thanks Kovid, I've now finally been able to download the epaper and put it on my reader with just one click. For the immense effort you must have put into building Calibre, and your continuing efforts to keep it up to date and provide support, I have sent a donation your way Keep up the great work!

Sander

Last edited by quisvir; 10-29-2009 at 02:08 PM.
quisvir is online now   Reply With Quote
Old 10-30-2009, 03:39 AM   #10
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Hello Sander,

Quote:
Originally Posted by quisvir View Post
Thanks Kovid, I've now finally been able to download the epaper and put it on my reader with just one click.
Sander
Could you explain how you did that?

Regards,

Joop
JvdW is offline   Reply With Quote
Old 10-30-2009, 09:50 AM   #11
quisvir
PRS+ contributor
quisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animals
 
quisvir's Avatar
 
Posts: 237
Karma: 6875
Join Date: Feb 2009
Location: Netherlands
Device: Sony 650 & T1
To be honest, in an extremely inefficient and rounabout way. Because I don't know any scripting or programming language, this is the entire process (batch file) in a nutshell:
- download epub using firefox with imacro plugin (I couldn't figure out how to use wget to download only the latest epub, considering that the filename changes every day), killing it when done
- use ebook-meta to add News tag to epub
- temporarily move current calibre db to different folder
- add epub to calibre with calibredb, forcing it to make a new db and give the epub id 1 (again, because I didn't know how to take the number from the calibredb list command )
- start calibre, which automatically uploads and deletes the epub
- kill calibre when this is done, and restore db folder

I can send you the script(s) I use if you want, but I imagine you'd prefer a slightly more efficient way of doing things
quisvir is online now   Reply With Quote
Old 11-02-2009, 08:19 AM   #12
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Quote:
Originally Posted by quisvir View Post
I can send you the script(s) I use if you want, but I imagine you'd prefer a slightly more efficient way of doing things
I'm close to a script which does it all.

Regards,

Joop
JvdW is offline   Reply With Quote
Old 11-02-2009, 10:11 AM   #13
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Quote:
Originally Posted by JvdW View Post
I'm close to a script which does it all.
OK, here is the batch file but one problem
There seems to be a 'bug' in Calibre where once you start one of Calibres executables in a batchfile that all other commands are ignored after the first Calibres executable finishes.

The batch file works since I have copied and pasted the commands to a DOS prompt and adjusted if needed, %%a needs to be %a.

Let me know if it doesn't work once the bug with Calibre is solved.

Happy

Joop
Attached Files
File Type: bat nrc.bat (2.4 KB, 111 views)
JvdW is offline   Reply With Quote
Old 11-03-2009, 12:27 PM   #14
quisvir
PRS+ contributor
quisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animalsquisvir is kind to children and small, furry animals
 
quisvir's Avatar
 
Posts: 237
Karma: 6875
Join Date: Feb 2009
Location: Netherlands
Device: Sony 650 & T1
Great work Joop!

I don't think there is a bug in Calibre, the only problem I've found is that the calibre-debug command doesn't work if Calibre is running.

I've made a few changes, the script now does this:
- First, it closes calibre. I don't know if calibre has a command line switch for this, so I just use tskill calibre. Optionally, you can add ebook-device eject to prevent data loss.
- Second, date variables are set (no change here, good work on it btw).
- Use Wget to download today's paper - removed some unnecessary parameters, fixed a few others.
- Add news tag, add to calibre database (no change here I think).
- Set calibre to auto-upload to reader (small change here).
- Clean up temporary files
- Start calibre in tray.

In other words, if you have Calibre running (either normally or in the background), the script still works. If it's running, it just kills it and starts it again once it is done.

There are two issues:
- Killing calibre like this is not optimal, I'd rather have the calibre-debug command work while calibre is still running so that killing/closing is not necessary.
- The script only works to download today's paper. So it doesn't work if you try to download after midnight, or for example the next morning. Also, it doesn't work on Sunday. The way I wanted to fix this is by making the script follow the link under the 'Download ePaper' button on the NRC site, so that it always downloads the latest paper. However, I've not (yet) been able to do this with wget.
Attached Files
File Type: bat nrc.bat (2.6 KB, 103 views)
quisvir is online now   Reply With Quote
Old 11-04-2009, 05:44 AM   #15
JvdW
Zealot
JvdW doesn't litterJvdW doesn't litter
 
Posts: 115
Karma: 146
Join Date: Jul 2008
Location: Netherlands Veenendaal
Device: Palm T5, Sony PRS-505, Nook Color
Quote:
Originally Posted by quisvir View Post
Great work Joop!

I don't think there is a bug in Calibre, the only problem I've found is that the calibre-debug command doesn't work if Calibre is running.
Maybe you're running a version older than 0.6.17 since I have had the problem with .19 and .20 and the person who found it can trace it back to .17.

Quote:
I've made a few changes, the script now does this:
- First, it closes calibre. I don't know if calibre has a command line switch for this, so I just use tskill calibre. Optionally, you can add ebook-device eject to prevent data loss.
- Second, date variables are set (no change here, good work on it btw).
- Use Wget to download today's paper - removed some unnecessary parameters, fixed a few others.
- Add news tag, add to calibre database (no change here I think).
- Set calibre to auto-upload to reader (small change here).
- Clean up temporary files
- Start calibre in tray.

In other words, if you have Calibre running (either normally or in the background), the script still works. If it's running, it just kills it and starts it again once it is done.

There are two issues:
- Killing calibre like this is not optimal, I'd rather have the calibre-debug command work while calibre is still running so that killing/closing is not necessary.
- The script only works to download today's paper. So it doesn't work if you try to download after midnight, or for example the next morning. Also, it doesn't work on Sunday. The way I wanted to fix this is by making the script follow the link under the 'Download ePaper' button on the NRC site, so that it always downloads the latest paper. However, I've not (yet) been able to do this with wget.
Nicely cleaned up, thanks.

Maybe its an idea to set a variable to indicate if Calibre was running when the script starts if it wasn't you'll end up with Calibre running when you maybe don't want that.

My idea was to schedule a task slightly past 15:00 on mo-fri and >07:00 at saturdays. That would take care of the what/when is TODAY.

After quite a bit of hacking and reading up I have come up with the following:

Code:
REM try to get the download link from the page and try to figure out what day it is for
for /f "tokens=1-10 delims==> """ %%a in ('findstr /i "Mousedown" nrc.tmp ^| findstr /i epubd') do set url="http://epaper.nrc.nl/%%~c=%%d=%%~e
for /f "tokens=1-10 delims==> """ %%a in ('findstr /i "Mousedown" nrc.tmp ^| findstr ID ^') do for /f "tokens=1 delims=&ft" %%i in ("%%d") do set datum=%%i

REM download the epaper of TODAY ;-)
REM wget --output-document=%TEMP%\nrc-%TODAY%.epub --load-cookies=%TEMP%\nrc.cookies "http://epaper.nrc.nl/downloadfile.php?ID=%TODAY%&ft=epubd"
wget --output-document=%TEMP%\nrc-%datum%.epub --load-cookies=%TEMP%\nrc.cookies %url%
I'm too sure about the stability of the for loops. If they change the website then it needs to be adapted. The previous solution will always work in combination with the right schedule.

Regards,

Joop
JvdW is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Automatic News Download brewjono Calibre 3 10-09-2010 08:49 AM
automatic lrf for news fetching? tuncaysu Calibre 2 01-15-2010 09:13 AM
automatic tag changes for news feeds in calibre? SDgirl Sony Reader 0 06-03-2009 09:11 PM
My automatic "feeds to Memory card" trick... sherman Calibre 0 11-22-2008 04:42 PM
Having trouble getting feeds to work; willing to help but don't know how tdavie Calibre 12 10-05-2008 08:01 PM


All times are GMT -4. The time now is 04:39 AM.


MobileRead.com is a privately owned, operated and funded community.