[GUI Plugin] FanFictionDownLoader

[susanpr]

Originally Posted by susanpr

I do, I'm totally at a loss as to what to try. Maybe delete the genetate cover plugin and reinstall?

That worked! I'm not sure what changed between Sunday and today to cause the whole mess, but at least its working now. Thank you so much for your help

[JimmXinu]

Originally Posted by susanpr

I do, I'm totally at a loss as to what to try. Maybe delete the genetate cover plugin and reinstall?

If you think the problem is Generate Cover, look in the FFDL config at the Generate Cover tab and make sure you don't have it configured to run GC for you.

If you delete Generate Cover and try FFDL without it installed, that will definitely tell you if GC is the problem.

[susanpr]

I Feel ;ike I'm jinxed. Now I get this message for everything I attempt to download:


Status
Title
Author
Comment
URL
Bad
A Working Title: Part Three
IBACULLEN
File contains no section headers.
file: <???>, line: 1
u' [epub]\n'
https://www.fanfiction.net/s/9396602/1/
Bad
Baby
HelliesWorld
File contains no section headers.
file: <???>, line: 1
u' [epub]\n'
https://www.fanfiction.net/s/10213016/1/
Bad
Barely Breathing
chocaholic123
File contains no section headers.
file: <???>, line: 1
u' [epub]\n'
https://www.fanfiction.net/s/10241842/1/

[JimmXinu]

Originally Posted by susanpr

I Feel ;ike I'm jinxed. Now I get this message for everything I attempt to download:

If you run calibre from a command prompt as

calibre-debug -g

... you'll get additional debug output in the command prompt window. Sometimes that contains additional useful information.

But this is really looking like something wrong with your computer or your install, not the software.

[skb]

Originally Posted by JimmXinu

The project source code is kept here, but a calibre plugin is just a zip of all the python code anyway.

To add a new adapter, you need to add a new file:

fanficdownloader/adapters/adapter_<sitename>.py

...and edit: fanficdownloader/adapters/__init__.py to import your new file.

Okay, got the code, edited the two files. Zipped up the code again and get the error message when adding the plugin (via Prefs/Advanced/Plugins):

calibre, version 1.33.0
ERROR: Unhandled exception: <b>InvalidPlugin</b>:The plugin in u'/Users/sue/Downloads/FanFictionDownLoaderPlugin_fictioning.zip' is invalid. It does not contain a top-level __init__.py file

calibre 1.33  isfrozen: True is64bit: True
Darwin-10.8.0-i386-64bit Darwin ('64bit', '')
('Darwin', '10.8.0', 'Darwin Kernel Version 10.8.0: Tue Jun  7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386')
Python 2.7.5
OSX: ('10.6.8', ('', '', ''), 'i386')
Traceback (most recent call last):
  File "site-packages/calibre/gui2/preferences/plugins.py", line 308, in add_plugin
  File "site-packages/calibre/customize/ui.py", line 375, in add_plugin
  File "site-packages/calibre/customize/ui.py", line 53, in load_plugin
  File "site-packages/calibre/customize/zipplugin.py", line 188, in load
  File "site-packages/calibre/customize/zipplugin.py", line 282, in _locate_code
InvalidPlugin: The plugin in u'/Users/sue/Downloads/FanFictionDownLoaderPlugin_fictioning.zip' is invalid. It does not contain a top-level __init__.py file

I've tried two ways...one download the source, edit the files, zip up and try to install and I downloaded your latest "beta" (on Post 1) and did the same. I get the same error both ways.

Calibre 1.33
Ver of plugin modified Version 1.8.18

I've added my modded files zip.

[Firedancer885]

Originally Posted by skb

I've tried two ways...one download the source, edit the files, zip up and try to install and I downloaded your latest "beta" (on Post 1) and did the same. I get the same error both ways.

Calibre 1.33
Ver of plugin modified Version 1.8.18

I've added my modded files zip.

you made a faulty zip archive. Look into an official ffdl plugin zip and modify the structure of your own to match.

[JimmXinu]

Originally Posted by skb

Okay, got the code, edited the two files. Zipped up the code again and get the error message when adding the plugin (via Prefs/Advanced/Plugins):

calibre, version 1.33.0
ERROR: Unhandled exception: <b>InvalidPlugin</b>:The plugin in u'/Users/sue/Downloads/FanFictionDownLoaderPlugin_fictioning.zip' is invalid. It does not contain a top-level __init__.py file
...

Just as the error says and Firedancer885 points out, the zip file isn't structured correctly--it has to have a __init__.py file in the top level. The contents need to be in the top level, not one down.

I don't know how easy zip files are to work with on mac, but in windows & linux it's easy to update files in an existing zip file. So you might try adding your files to the official zip.

Originally Posted by skb

I've tried two ways...one download the source, edit the files, zip up and try to install and I downloaded your latest "beta" (on Post 1) and did the same. I get the same error both ways.
...

If you downloaded the source from the project site, there's a python program included specifically to make the zip file correctly. From the top source directory, run:

python makeplugin.py

That will also leave out some unnecessary files and reduce the size of the zip by ~100k.

[skb]

Originally Posted by JimmXinu

If you downloaded the source from the project site, there's a python program included specifically to make the zip file correctly. From the top source directory, run:

python makeplugin.py

That will also leave out some unnecessary files and reduce the size of the zip by ~100k.

Well that's embarrassing. Sigh. I downloaded the cooked plugin not the source. Newbies, eh? We're a menace.

The good news is that I finally downloaded the correct files (yay) and added/edited the two files (yay) and even compiled the plugin using Terminal (after swearing in 2002 that I'd never use Linux/Unix again. Darn... but yay).

Installed with no issues.

However, when I try to download a story from my site, I get the error:

Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
'NoneType' object has no attribute 'strftime'
http://www.fictioning.net/viewstory.php?sid=503

So close and yet so far.

Edited: the good news is though I can change the fictioning.net site if there's a problem there, as I "own" it. But I'm not sure what I'd have to change...

[JimmXinu]

Originally Posted by skb

'NoneType' object has no attribute 'strftime'
http://www.fictioning.net/viewstory.php?sid=503

I recently posted about running in debug mode. That will tell you where the error is.

But experience tells me that you're probably calling story.setMetadata for dateUpdated with a None value.

When you get this adapter working, I'd be willing to add it to FFDL if you like. Just because I don't want to do small sites doesn't mean other people can't.

[skb]

Originally Posted by JimmXinu

I recently posted about running in debug mode. That will tell you where the error is.

Thanks - I tried to find in the GUI how to set debug (unix-avoidance again).

Anyway, I seem to be throwing this error

FFDL:ERROR:calibre_plugins.fanfictiondownloader_plugin.dialogs(600):Exception: {u'comment': u"'NoneType' object has no attribute 'strftime'", u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'http://www.fictioning.net/viewstory.php?sid=434', u'listorder': 0, u'author_sort': [u'Unknown'], u'site': 'www.fictioning.net', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'strftime'

So, does that mean I have issues in the Comments & Author areas?

[JimmXinu]

Originally Posted by skb

Thanks - I tried to find in the GUI how to set debug (unix-avoidance again).

Despite being a linux lover for all my real job work, I do all my calibre development on windows. Windows does have a command prompt, too.

Originally Posted by skb

Anyway, I seem to be throwing this error

FFDL:ERROR:calibre_plugins.fanfictiondownloader_plugin.dialogs(600):Exception: {u'comment': u"'NoneType' object has no attribute 'strftime'", u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'http://www.fictioning.net/viewstory.php?sid=434', u'listorder': 0, u'author_sort': [u'Unknown'], u'site': 'www.fictioning.net', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'strftime'

So, does that mean I have issues in the Comments & Author areas?

I don't know why you'd think that. That's the same error messages as before (no additional debug info) and it's saying it's tried to call the datetime string formatting routine (strftime) on a None (python's version of null or nil) where there should be a datetime object.

[skb]

I gave up the DOS prompt at the same time. It's not really working out for me, though, is it? But actually, I too love linux (so stable!) but these days need a GUI because I'm old & grumpy and though I code HTML, I'm not really a developer, by any stretch of the imagination.

DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(156):Plugin FanFictionDownLoader macmenuhack file_path:/Users/sue/Library/Preferences/calibre/plugins/fanfictiondownloader_macmenuhack.txt
macmenuhack file_path:/Users/sue/Library/Preferences/calibre/plugins/fanfictiondownloader_macmenuhack.txt
DEBUG:    3.4 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    3.4 rebuild_menus - self.supports_ratings=False
Started up in 9.97 seconds with 3292 books
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(726):FanFictionDownLoader v1.8.18
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin(761):url:http://www.fictioning.net/viewstory.php?sid=434
FFDL:DEBUG:calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.adapter_fictioningnet(68):URL: http://www.fictioning.net/viewstory.php?sid=434&index=1
FFDL:ERROR:calibre_plugins.fanfictiondownloader_plugin.dialogs(600):Exception: {u'comment': u"'NoneType' object has no attribute 'strftime'", u'begin': None, u'good': False, u'end': None, u'author': [u'Unknown'], u'url': u'http://www.fictioning.net/viewstory.php?sid=434', u'listorder': 0, u'author_sort': [u'Unknown'], u'site': 'www.fictioning.net', u'comments': u'', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'strftime'
Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.dialogs", line 590, in do_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 839, in prep_download_loop
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.adapters.base_adapter", line 261, in getStoryMetadataOnly
  File "calibre_plugins.fanfictiondownloader_plugin.fanficdownloader.story", line 269, in setMetadata
AttributeError: 'NoneType' object has no attribute 'strftime'

That's what I get on the console when I try to download a story. The only bit I truly understood is that I have 3000+ books. Yikes!

I have NO idea what I'm doing. Okay, so it's a date/time error. So, I'm guessing I have an issue with something like updated or published date. I'll investigate & experiment further.

Thanks for all the help. If I get through this alive, I might put together an "Idiot's Guide" procedure (let's face it, I'm qualified).

[skb]

Hmm, well, I truly have "fixed" it now. Now when I try to add a story I get "No valid story URLs entered".

Could this particular thing be a permissions issue or even a file encoding type error? I ask this because when I compile the plugin, all the files have the same date except the ones I edited/created. Does this matter. Could it be a Win/Unix file format thing (I always made that error in the old days).

Edited: Okay, getting somewhere now. Compiled again. now getting

object of type 'NoneType' has no len()

But, I cannot get debug to work again to give the full error. Still...getting there.

[Firedancer885]

Is there some kind of adapter how-to? I tried to put together an adapter for an efiction archive and kept running into different combinations of 'NoneType' object has no attribute 'strftime' and 'NoneType' object has no attribute __getitem__' depending on the skin I viewed the site with. After a few hours I gave up. I have some experience with pascal, python and coding, but this type of thing (adapter writing) is very new to me.

[JimmXinu]

For serious debugging of adapters, I use the CLI (command line interface) of FFDL and skip calibre all together. That way I'm not recreating the zip and restarting calibre every time I change something.

I'm on windows, so I have a run.bat file containing:

set PYTHONIOENCODING=UTF-8
c:\Python27\python.exe downloader.py -d %1 %2 %3 %4 %5 %6 %7 %8 %9

Then I just do 'run <url>'. downloader.py has other options as well, like -m for metadata only.

I also use lots of print statements (or logger.debug()) for debugging. The errors skb & Firedancer885 are reporting from their adapters are from not collecting or setting all the appropriate metadata.

If the track traces aren't enough to tell you which metadata you're not setting, I'd use print statements in the adapter to show what metadata you are setting.