Auto-delete news

[chaley]

Given that the auto-connect feature is used primarily to download news and given requests such as NSILMike's, I decided to implement automatic deleting of news "books" after some time, matching what calibre does. If a book has the tag "News" and the author "calibre" (both case sensitive) then it is a candidate for deletion.

There are three new preferences:

• A checkbox to turn on auto-deletion
• A text preference to specify the "News" tag. This is necessary because in calibre the tag is controled by the language setting.
• The number of days to wait. The deletion process runs at midnight, and if at that instant the book is older than this number of days then it will be deleted. This means that a book will persist from N to almost N+1 days.

The preferences screens are shown below.

I intend to release this feature to beta this weekend.

Comments?

[NSILMike]

Appreciate your effort very much! Some comments and help(?)
1) How does one tell if a specific book has the 'news' tag? In Calibre, I right click the book, and choose 'edit metadata'
2) If I am reading your 3rd bullet correctly, I think to delete yesterday's news at midnight I would set N = 1. What would happen if I set it to 2, so that early the next day I can read things like news blogs that update late in the day, so are set in Calibre to download at 5PM (for example?)
3) Can N be set to different values for individual items? For instance if I use Calibre to download and sync a weekly or monthly magazine, how would that work?

[chaley]

Originally Posted by NSILMike

Appreciate your effort very much! Some comments and help(?) for the beta testers...
1) How does one tell if a specific book has the 'news' tag? In Calibre, I right click the book, and choose 'edit metadata'

Look at the tags in book details. If this is something you want to know easily for all books then you could add tags to the book list display. Finally, you can tap the "News" tag in the grouping drawer to see all books with that tag.

2) If I am reading your 3rd bullet correctly, I think to delete yesterday's news at midnight I would set N = 1.

Yes. If you set it to zero, all news will be deleted at midnight. If you set it to 1, the "book" must be at least 24 hours old at midnight. And so on.

3) Can N be set to multiple values? For instance if I use Calibre to download and sync a weekly or monthly magazine, how would that work?

No. That would require using a different N based on some tag or book title pattern, which is far further than I want to go.

[chaley]

Originally Posted by NSILMike

Can N be set to different values for individual items? For instance if I use Calibre to download and sync a weekly or monthly magazine, how would that work?

Thinking about this, I can do something that pushes the complexity back to you but would solve the problem.

When you schedule a news download, calibre lets you add extra tags to the news source that will be applied to every book. My thought is that you could add a special tag that tells CC how long news books downloaded by this source should remain on the device. Possible example: cc_delete_after_N where N is the number of days for this news source.

CC would process the news normally using the "master" days number. No book would be deleted if it is younger than this value. CC would then check for the existence of a tag beginning with "cc_delete_after_". If CC finds such a tag it would strip the number off the end and delete the book only if that many days (actually (number * 24) hours) had passed.

[NSILMike]

that works!

[PeterT]

I like this feature! Thumbs up!!

[Purple Lady]

With the addition to be able to delete different feeds at different times, I will use this. I have both a news feed and comics feeds, but sometimes don't read the comics every day. News might get old, but comics never do!

[chaley]

I have added the cc_delete_after_N tag processing. With that each feed can have its own timeout, subject to the minimum in CC's settings. If you want the tag to have total control then set CC's preference to zero.

[NSILMike]

So far, it's perfect. I've been setting daily news to zero (and haven't had enough time to test weekly or monthly items yet.) If I set daily to one, then two downloads would co-exist in Companion on my device, right? Could/Would that cause any issues that I'd need to look out for...?

[chaley]

Originally Posted by NSILMike

So far, it's perfect. I've been setting daily news to zero (and haven't had enough time to test weekly or monthly items yet.) If I set daily to one, then two downloads would co-exist in Companion on my device, right? Could/Would that cause any issues that I'd need to look out for...?

Thanks for the feedback!

Yes, if your default setting is 1 day then you can have two issues from a given daily news source in CC. I can be more specific and say that you will have two copies of daily news sources if you use auto-connect and the auto-connect is after 00:01. Because in CC a day is 24 hours, with a default of 1 day a news book won't be a candidate for deletion until the second midnight after it is downloaded.

I don't see why this situation would cause a problem. They have different file names because the date is embedded into the path. I did exactly this, had multiple copies of a news source, during testing and it worked fine.

[NSILMike]

Originally Posted by chaley

Thanks for the feedback!

Yes, if your default setting is 1 day then you can have two issues from a given daily news source in CC. I can be more specific and say that you will have two copies of daily news sources if you use auto-connect and the auto-connect is after 00:01. Because in CC a day is 24 hours, with a default of 1 day a news book won't be a candidate for deletion until the second midnight after it is downloaded.

I don't see why this situation would cause a problem. They have different file names because the date is embedded into the path. I did exactly this, had multiple copies of a news source, during testing and it worked fine.

I'm not getting two issues... This is interesting. If I leave a news source on my tablet, and a new one downloads via autoconnect, I only get one issue. And the weird part is the content is from the new download but the 'Added to CC' tag is from the prior issue. For instance, I read CNET. As of right now, the 10 June issue is in Calibre. In CC, the 'added to calibre' tag is correct (10 Jun 2015) but the Added to CC is 2015-06-09

[chaley]

Originally Posted by NSILMike

I'm not getting two issues... This is interesting. If I leave a news source on my tablet, and a new one downloads via autoconnect, I only get one issue. And the weird part is the content is from the new download but the 'Added to CC' tag is from the prior issue. For instance, I read CNET. As of right now, the 10 June issue is in Calibre. In CC, the 'added to calibre' tag is correct (10 Jun 2015) but the Added to CC is 2015-06-09

My assumption was that the news source generated the "books" with a unique file name incorporating the date. If it does not or of tje dates are not sufficiently unique then all bets are off. CC will think that it is an edit of a previous book and probably get quite confused. For example, the "last modified" date for the book might not change, so when the new "book" is sent the new metadata might not be sent with it.

If you look in CC's Book Details, does the file name for the news book contain a date? Is a new book created when a subsequent news edition is downloaded?

[NSILMike]

Originally Posted by chaley

My assumption was that the news source generated the "books" with a unique file name incorporating the date. If it does not or of tje dates are not sufficiently unique then all bets are off. CC will think that it is an edit of a previous book and probably get quite confused. For example, the "last modified" date for the book might not change, so when the new "book" is sent the new metadata might not be sent with it.

If you look in CC's Book Details, does the file name for the news book contain a date? Is a new book created when a subsequent news edition is downloaded?

Not sure where to see the file name in CC's Book Details, so I looked at the specific file name in 'ES File Explorer' which allows me to navigate to the path where the files are stored in Flash on my Nexus tablet. The file name for each and every new source I have downloaded contains a date. For instance in this case (CNET) the file name is CNET News_2015-6-9.mobi All the others are similar... News source name followed by the date.

Update: If I go to 'choose items to display' in CC's settings and set the path to be displayed, I get the same file name (preceded by the path in storage of course...)

[chaley]

Originally Posted by NSILMike

Not sure where to see the file name in CC's Book Details, so I looked at the specific file name in 'ES File Explorer' which allows me to navigate to the path where the files are stored in Flash on my Nexus tablet. The file name for each and every new source I have downloaded contains a date. For instance in this case (CNET) the file name is CNET News_2015-6-9.mobi All the others are similar... News source name followed by the date.

Update: If I go to 'choose items to display' in CC's settings and set the path to be displayed, I get the same file name (preceded by the path in storage of course...)

The file has the date 9 June, not 10 June. That is the root of the problem. And given that calibre assigns file names, I don't see what CC can do about it.

Looking at calibre's code, in some (perhaps all) situations it sets the pubdate to the current time. The date used in the file name comes from the pubdate. This could lead to interesting consequences if a news article for day X is downloaded in a timezone with the current time having day X-1 or X+1. Calibre's pubdate will not match the news "day".

What it looks like is that two CNET downloads happened on the same localtime day, perhaps one just after midnight and one just before midnight. That would give them the same file name but different metadata. Because the file names are the same, CC thinks that the book is not new. The "added to CC" date will not change, although everything else about the book might.

So, to end this flight of speculation, when does calibre download the news? Is the date in the news file name always one day old? Does the newest edition always overwrite the previous edition?

[NSILMike]

Originally Posted by chaley

The file has the date 9 June, not 10 June. That is the root of the problem. And given that calibre assigns file names, I don't see what CC can do about it.
[snipped]
So, to end this flight of speculation, when does calibre download the news? Is the date in the news file name always one day old? Does the newest edition always overwrite the previous edition?

1) Yes, and that seems strange to me since it is clearly downloaded on the 10th, and Calibre correctly deletes the prior edition from the 9th.
2) I have windows set to start Calibre automatically at 4PM each day, and Calibre is set to download several news sources each day at slightly staggered times between 4PM and 5:30PM. Many are set to cc_delete_after_0, but a few are set to cc_delete_after_1. In this specific case (CNET) it's set to 1. This is not much of an issue, so if difficult to fix (as it apparently is...) in CC, I'm fine with it as is. I am getting the updated news source which is my primary objective, and I could just default to cc_delete_after_0 for the daily items. Many thanks for your efforts.