[GUI Plugin] OPML Importer (RSS feeds from feedly/google reader)

[fem]

This plugin has been assimilated into calibre as from version 1.32! No need for you to download this anylonger

I didn't want to manually import my 100+ RSS feeds from feedly, so I wrote this plugin.

The project page is located here: https://github.com/ingkebil/OPML2Recipe

Feel free to report issues and feature requests, and donate. 50% goes to the Free Software Foundation Europe.

Main Features

Import RSS feeds as an OPML file from feedly or google reader.
This plugin will create a recipe for each group and each remaining feed.
So far a restart is required to have the imported RSS feeds show up in custom news sources.

Version History

Spoiler:

* Version 1.0.1: Added a MessageBox when the conversion was succesful.
* Version 1.0.0: Created a working import plugin + configuration of oldest article and max articles.

Special Notes

Requires calibre version 0.7.53 or later. This plugin has been assimilated into the main calibre code base as from version 1.32!
A restart is still required once the RSS feeds have been imported!
I forgot to add a 'Import finished' message box when the import completes, that's for tomorrow.

Installation Notes

Download the attached zip file, install the plugin and restart calibre as described in the Introduction to plugins thread.
Customize the plugin via Preferences -> Plugins or through the configure button of the plugin.

Usage

Set the max articles and oldest article preferences. If not set, the defaults will be used.
Hit the 'import' button, select the OPML files you want to import.
Restart calibre to have the imported RSS feeds show up in custom news sources.

Attached Files

[roen.v.touli]

Imported feeds just fine, however, I am unable to edit them afterward. When I click on a recipe in the 'add custom news source' dialogue I get the following error:

calibre 1.25 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Traceback (most recent call last):
File "site-packages\calibre\gui2\dialogs\user_profiles.py", line 167, in current_changed
File "site-packages\calibre\gui2\dialogs\user_profiles.py", line 358, in populate_options
TypeError: QSpinBox.setValue(int): argument 1 has unexpected type 'long'

Other than that, great plugin! I've been patiently waiting for one like this as I have hundreds of feeds. It would be nice if you might be able to make it where before or after import, you can choose to to combine recipe's, change feed name, add tags. Future wish: a scrolling ticker integrated into the plugin and Calibre.

Keep up the good work! Thank you.

[fem]

Originally Posted by roen.v.touli

It would be nice if you might be able to make it where before or after import, you can choose to to combine recipe's, change feed name, add tags. Future wish: a scrolling ticker integrated into the plugin and Calibre.

I gave this some thought while developing the plugin. Ideally, this functionality shouldn't be a plugin at all, but should be made available on the custom news souces dialog. This way all the power already implemented for that dialog (which mainly is name collision detection when importing feeds) would be made available out of the box. Added bonus: restart to import the feeds into calibre could be made redundant.

I'll look into the error tonight.

[roen.v.touli]

Originally Posted by fem

I gave this some thought while developing the plugin. Ideally, this functionality shouldn't be a plugin at all, but should be made available on the custom news souces dialog. This way all the power already implemented for that dialog (which mainly is name collision detection when importing feeds) would be made available out of the box. Added bonus: restart to import the feeds into calibre could be made redundant.

I'll look into the error tonight.

I agree this should be native to Calibre, however, I know Kovid is very busy on the project with lots on his plate. I am patient that News will have better functionality and that this will be done in the future. Lack of functionality with News is the only thing keeping me with a separate RSS aggregator.

Thanks for looking into the error fem.

[eschwartz]

Originally Posted by roen.v.touli

I agree this should be native to Calibre, however, I know Kovid is very busy on the project with lots on his plate. I am patient that News will have better functionality and that this will be done in the future. Lack of functionality with News is the only thing keeping me with a separate RSS aggregator.

Thanks for looking into the error fem.

Kovid is never too busy to accept patches, it takes less effort to check the effects of existing code than to come up with new code.

@fem, if you integrate this into the main calibre codebase, I'm sure Kovid would use it. It sounds like something that deserves to be built-in.

[fem]

Originally Posted by roen.v.touli

Imported feeds just fine, however, I am unable to edit them afterward. When I click on a recipe in the 'add custom news source' dialogue I get the following error:

calibre 1.25 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Traceback (most recent call last):
File "site-packages\calibre\gui2\dialogs\user_profiles.py", line 167, in current_changed
File "site-packages\calibre\gui2\dialogs\user_profiles.py", line 358, in populate_options
TypeError: QSpinBox.setValue(int): argument 1 has unexpected type 'long'

So I've tested this on the ubuntu 64bit and win7 64bit version of calibre and
I can't seem to reproduce the error. Any chance you can upload one of the resulting recipes or (a partial) OPML file that causes this error?

[roen.v.touli]

Originally Posted by fem

So I've tested this on the ubuntu 64bit and win7 64bit version of calibre and
I can't seem to reproduce the error. Any chance you can upload one of the resulting recipes or (a partial) OPML file that causes this error?

Attached is a zip with several of my recipes and the OPML in entirety for better testing. I am wondering if there are too many recipes that is overloading Calibre? Quite a few of my RSS feeds are already recipes built-in, I am just not wanting to go item by item finding duplicates...but I might have to?

Calibre was not updated as I was waiting for tomorrow but I updated to make sure that was not the cause and the error still persists.

Hope this helps and that the entire OPML is ok.

[fem]

The problem seems to be that the recipes have oldest_article and max_articles_per_feed set to a number with 34 digits, instead of something more reasonable like 7 and 100. I don't really know how that happened tho , as invalid values would be caught before a recipe is actually compiled and written to disk.

A quick fix would be to delete the imported recipes and reimport them making sure actual valid values have been entered in the max articles and oldest article fields. Even going so far as to just increase/decrease those values regardless of valid values being displayed.
You'll prolly want to find calibre's configuration directory and delete the last added recipes from there instead of manually going over them in the custom news source dialog.

If removing the feeds would provide too much of a hassle, I could add a 'remove imported feeds' button to the plugin to easy the pain of removing the recipes.

Let me know if this helps.

[roen.v.touli]

Originally Posted by fem

The problem seems to be that the recipes have oldest_article and max_articles_per_feed set to a number with 34 digits, instead of something more reasonable like 7 and 100. I don't really know how that happened tho , as invalid values would be caught before a recipe is actually compiled and written to disk.

A quick fix would be to delete the imported recipes and reimport them making sure actual valid values have been entered in the max articles and oldest article fields. Even going so far as to just increase/decrease those values regardless of valid values being displayed.
You'll prolly want to find calibre's configuration directory and delete the last added recipes from there instead of manually going over them in the custom news source dialog.

If removing the feeds would provide too much of a hassle, I could add a 'remove imported feeds' button to the plugin to easy the pain of removing the recipes.

Let me know if this helps.

Worked perfectly, thank you! I tried to set each of those to maximum in 'configure' but did not realize I put in so many digits. At least we now know the plugin is not catching invalid values.

I think most of us on this forum are familiar enough with Calibre to go into its directory and delete the files there; however, for new users, ease of access, and simplicity, it may not be a bad idea to have a delete feeds button where you could select multiple feeds and delete at once.

I see that Kovid does not mind you developing this into Calibre native system itself...can't wait to see how it goes. Any ideas or thoughts of what you are going to do? I think inspiration from current RSS readers and features they offer would be beneficial.

[fem]

So, in the meanwhile, Kovid and I wrote this plugin into the calibre main code base. As from version 1.32, you have native RSS feed import support through OPML files.
No restart is required, recipes show up instantaneously. It has been made easier to overwrite already existing recipes. But deleting recipes has not been touched and neither did the recipe interface change.

[CleverClogs]

Hi Kovid, Fem and others here in this thread,

It's very cool to see that you integrated Fem's plug-in into the core build of Calibre. OPML is much under discussion at the moment. The current approach of offering an import function unfortunately results in a static copy of that OPML file. I propose supporting remotely hosted OPML reading lists. These could be reading lists on someone's own website, on a feed reader's website, or even on a cloud service.
A few weeks I wrote an in-depth article about this topic that you may find of interest. It offers an introduction into the issues and then explains how one RSS feed reader, InoReader, resolved them: RSS Reader InoReader to Support Dynamic OPML Subscriptions, by Marjolein Hoekstra (@CleverClogs)

Marjolein Hoekstra
@CleverClogs

[fem]

Hah! This looks pretty interesting.

It should be rather easy to add the similar GUI options to Calibre as in InoReader:
* possiblity to add URL instead of File Upload text box
* checkbox to be able to check for a Full Synchronization

And then everytime before news is downloaded a synchronization could be performed.

I might have time later this week to take a deeper look into this. Doesn't seem like a hell of a lot of work.

[CleverClogs]

Hi Fem,

I'm glad you like the suggestion. Over the past few weeks I've been collecting a number of web-based OPML files that may come in handy while testing the new features. You can find them in this publicly editable, multi-tab Google spreadsheet.

Would it be possible to add an option that notifies a user of a change? Ideally this would be configurable, so that you can turn off notifications for very productive OPML files.

Thanks for keeping me posted!

[fem]

Hey CleverClogs,

really sorry for coming back to you so late. I'm in the middle of moving to Sweden, so I've had no time to look into this. I can't give you a time frame, as the next 2 months are going to be really busy.

All suggestions are reasonable. I do agree that it would be nicer if the user would be notified when an OPML file is updated, so I'll look into how this could be done nicely.

I'll keep you informed.

[CleverClogs]

Thanks for the update, Fem.
These must be exciting times for you. I hope everything goes smooth with your move.
Just ping me when you are ready.