Automatic news feeds uploading, How does it work?

[JvdW]

Hello Kovid,

Atleast two people, , are interested in how you detect/keep track of what needs to be uploaded to a connected reader.
For example:
If I download some newsfeeds and have a look in the database then I can't find anything different from regular books that are in there. Still when i connect my reader those feedbooks will be uploaded and deleted, if that preference is set.
I had a look at the source but this is beyond my python skills to find out myself.

The reason I'm curious is that it would be nice to be able to add an epub of a newspaper through calibre-db.exe and have that automatically uploaded when the reader connects.

Could you explain or give a pointer to the relevant part of the calibre source.

Thanks,

Joop

[kovidgoyal]

Uses the tag News (or the translation of it in the local language)

[JvdW]

Originally Posted by kovidgoyal

Uses the tag News (or the translation of it in the local language)

I'm pretty sure I tried that because that seemed logical todo. I'll give it another try tomorrow morning. (GMT+1)

Thanks,

Joop

[veysey]

I have tried to do this too, using the tag news, but in conjunction with other tags as well. e.g. "News, Humor". Did not seem to automatically upload. Will this only work for "pure" news feeds?

Cheers,

John

[kovidgoyal]

Actually, I'm sorry, what happens is that when a news download completes, the id is recorded in a config file which is then used to sync

Assuming you can find out the id of the freshly added news (which should be easy to do using calibredb, the following command will mark it to be synced

calibre-debug -c "from calibre.gui2 import dynamic; ids = dynamic.get('news_to_be_synced', set([])); ids.add(35); dynamic.set('news_to_be_synced', ids)"

replace 35 with the id of the freshly added news

[JvdW]

Originally Posted by kovidgoyal

Actually, I'm sorry, what happens is that when a news download completes, the id is recorded in a config file which is then used to sync

Assuming you can find out the id of the freshly added news (which should be easy to do using calibredb, the following command will mark it to be synced

calibre-debug -c "from calibre.gui2 import dynamic; ids = dynamic.get('news_to_be_synced', set([])); ids.add(35); dynamic.set('news_to_be_synced', ids)"

replace 35 with the id of the freshly added news

Yes, that works provided that Calibre itself is not running when you run this commandline.

Ofcourse Calibre knows its a newsfeed because its tagged News, what I'm wondering about is if it might be possible to drop that requirement?

Regards,

Joop

[ankit]

A somewhat related question: is it possible for Calibre to delete the old version of the corresponding news from the device? I would like to keep only the current news version on the device. Checking to see if something like this is already hiding in the options somewhere.

[kovidgoyal]

syncing is on the TODO list. When it is implemented, all these things become automatic.

[quisvir]

Thanks Kovid, I've now finally been able to download the epaper and put it on my reader with just one click. For the immense effort you must have put into building Calibre, and your continuing efforts to keep it up to date and provide support, I have sent a donation your way Keep up the great work!

Sander

[JvdW]

Hello Sander,

Originally Posted by quisvir

Thanks Kovid, I've now finally been able to download the epaper and put it on my reader with just one click.
Sander

Could you explain how you did that?

Regards,

Joop

[quisvir]

To be honest, in an extremely inefficient and rounabout way. Because I don't know any scripting or programming language, this is the entire process (batch file) in a nutshell:
- download epub using firefox with imacro plugin (I couldn't figure out how to use wget to download only the latest epub, considering that the filename changes every day), killing it when done
- use ebook-meta to add News tag to epub
- temporarily move current calibre db to different folder
- add epub to calibre with calibredb, forcing it to make a new db and give the epub id 1 (again, because I didn't know how to take the number from the calibredb list command )
- start calibre, which automatically uploads and deletes the epub
- kill calibre when this is done, and restore db folder

I can send you the script(s) I use if you want, but I imagine you'd prefer a slightly more efficient way of doing things

[JvdW]

Originally Posted by quisvir

I can send you the script(s) I use if you want, but I imagine you'd prefer a slightly more efficient way of doing things

I'm close to a script which does it all.

Regards,

Joop

[JvdW]

Originally Posted by JvdW

I'm close to a script which does it all.

OK, here is the batch file but one problem
There seems to be a 'bug' in Calibre where once you start one of Calibres executables in a batchfile that all other commands are ignored after the first Calibres executable finishes.

The batch file works since I have copied and pasted the commands to a DOS prompt and adjusted if needed, %%a needs to be %a.

Let me know if it doesn't work once the bug with Calibre is solved.

Happy

Joop

[quisvir]

Great work Joop!

I don't think there is a bug in Calibre, the only problem I've found is that the calibre-debug command doesn't work if Calibre is running.

I've made a few changes, the script now does this:
- First, it closes calibre. I don't know if calibre has a command line switch for this, so I just use tskill calibre. Optionally, you can add ebook-device eject to prevent data loss.
- Second, date variables are set (no change here, good work on it btw).
- Use Wget to download today's paper - removed some unnecessary parameters, fixed a few others.
- Add news tag, add to calibre database (no change here I think).
- Set calibre to auto-upload to reader (small change here).
- Clean up temporary files
- Start calibre in tray.

In other words, if you have Calibre running (either normally or in the background), the script still works. If it's running, it just kills it and starts it again once it is done.

There are two issues:
- Killing calibre like this is not optimal, I'd rather have the calibre-debug command work while calibre is still running so that killing/closing is not necessary.
- The script only works to download today's paper. So it doesn't work if you try to download after midnight, or for example the next morning. Also, it doesn't work on Sunday. The way I wanted to fix this is by making the script follow the link under the 'Download ePaper' button on the NRC site, so that it always downloads the latest paper. However, I've not (yet) been able to do this with wget.

[JvdW]

Originally Posted by quisvir

Great work Joop!

I don't think there is a bug in Calibre, the only problem I've found is that the calibre-debug command doesn't work if Calibre is running.

Maybe you're running a version older than 0.6.17 since I have had the problem with .19 and .20 and the person who found it can trace it back to .17.

I've made a few changes, the script now does this:
- First, it closes calibre. I don't know if calibre has a command line switch for this, so I just use tskill calibre. Optionally, you can add ebook-device eject to prevent data loss.
- Second, date variables are set (no change here, good work on it btw).
- Use Wget to download today's paper - removed some unnecessary parameters, fixed a few others.
- Add news tag, add to calibre database (no change here I think).
- Set calibre to auto-upload to reader (small change here).
- Clean up temporary files
- Start calibre in tray.

In other words, if you have Calibre running (either normally or in the background), the script still works. If it's running, it just kills it and starts it again once it is done.

There are two issues:
- Killing calibre like this is not optimal, I'd rather have the calibre-debug command work while calibre is still running so that killing/closing is not necessary.
- The script only works to download today's paper. So it doesn't work if you try to download after midnight, or for example the next morning. Also, it doesn't work on Sunday. The way I wanted to fix this is by making the script follow the link under the 'Download ePaper' button on the NRC site, so that it always downloads the latest paper. However, I've not (yet) been able to do this with wget.

Nicely cleaned up, thanks.

Maybe its an idea to set a variable to indicate if Calibre was running when the script starts if it wasn't you'll end up with Calibre running when you maybe don't want that.

My idea was to schedule a task slightly past 15:00 on mo-fri and >07:00 at saturdays. That would take care of the what/when is TODAY.

After quite a bit of hacking and reading up I have come up with the following:

REM try to get the download link from the page and try to figure out what day it is for
for /f "tokens=1-10 delims==> """ %%a in ('findstr /i "Mousedown" nrc.tmp ^| findstr /i epubd') do set url="http://epaper.nrc.nl/%%~c=%%d=%%~e
for /f "tokens=1-10 delims==> """ %%a in ('findstr /i "Mousedown" nrc.tmp ^| findstr ID ^') do for /f "tokens=1 delims=&ft" %%i in ("%%d") do set datum=%%i

REM download the epaper of TODAY ;-)
REM wget --output-document=%TEMP%\nrc-%TODAY%.epub --load-cookies=%TEMP%\nrc.cookies "http://epaper.nrc.nl/downloadfile.php?ID=%TODAY%&ft=epubd"
wget --output-document=%TEMP%\nrc-%datum%.epub --load-cookies=%TEMP%\nrc.cookies %url%

I'm too sure about the stability of the for loops. If they change the website then it needs to be adapted. The previous solution will always work in combination with the right schedule.

Regards,

Joop