View Full Version : [GUI Plugin] FanFictionDownLoader


Pages : [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

JimmXinu
12-30-2011, 03:32 PM
This plugin has been replaced. Its replacement, FanFicFare (http://www.mobileread.com/forums/showthread.php?t=259221) contains all the same features plus new sites, features and fixes.

Aleyst
12-31-2011, 12:24 AM
Thanks for this. It's pretty handy.

JimmXinu
01-06-2012, 12:47 AM
Posting FanFictionDownLoader plugin 1.1.0, hoping to get some beta testers before pushing it out.

Internally, 1.1 is an almost complete re-write of the plugin from 1.0.

I was advised by more experienced plugin developers that the way I was handling background processing was not particularly safe. Given the number of unexpected crashes I kept getting working with 1.0, I saw their point.

1.1 uses Calibre's mechanism to launch separate background processes to download stories rather than using a background thread.

There is now a drop down menu from the plugin tool bar button for Add/Update/Configure/About.

Also, 'update' truly updates the book(s) selected now. Before the plugin gathered URLs from the selected books, but after that treated the URL as if the user had entered it. If the name or author on the book had been changed, either in Calibre or on the website, it would not update the existing book in Calibre, but treat it as a different book.

The plugin now downloads the stories, but does *not* add/update them in your library until all are downloaded and you confirm you want the updates. Again, this is because background processing updating the foreground is dangerous in Calibre.

There is (hopefully) better download error/update reporting now, but it could still use some more work.

There are more options now for how to update existing stories, whether by directly updating or by doing 'Add New' with a URL to a story that already exists in your library.

Under "Add New from URL(s)", there's an option to choose handling for "If Story Already Exists?". This is also sometimes called collision handling.

"Skip" and "Add New Book" should be self-explanatory.

"Update EPUB if New Chapters" and "Update EPUB Always" allow you to incrementally update a story you already have, but only if you use EPUB. "Update EPUB Always" will update the title page and metadata for the story even if there are no new chapters.

"Overwrite Always" will download the story and overwrite the chosen format. "Overwrite if Newer", only if the website story is newer.

"Update Calibre Metadata Only" does not download the story, only enough metadata to update the metadata Calibre keeps.

Update Existing's "Update Mode" is identical.

Under Configuration, there are some changes and new options.

With the split between "Add New" and "Update Existing", the options to take URLs from selected and prioritize selected vs clipboard are removed. There's still an option to take URLs from clipboard or not.

You can also make the menubar button always bring up "Add New" by unchecking "Default to Update when books selected?".

Jim

(Attachment removed after 1.1.1 posted.)

Aleyst
01-06-2012, 02:47 AM
Sounds interesting, I'm just updating the main calibre program now, I'll update the plugin and have a mess around. :D

JimmXinu
01-08-2012, 03:21 PM
1.1.1 contains everything from 1.1.0 plus:


Fixes for problems found with Add New.
Fix to 'delete other formats' when html.
Hide 'update epub' options when format not epub.
fanfic.castletv.net changed to castlefans.org/fanfic.
Fix for twiwrite adult-only stories.


I think this is ready to be pushed out as the new version, but I'd appreciate it if I could get a tester or two to try it first.

Jim

scoopaz
01-08-2012, 03:37 PM
Hi Jim,

Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 340, in dispatch
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 544, in download_list_completed
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 589, in _add_or_update_book
File "site-packages\calibre\library\database2.py", line 3175, in create_book_entry
File "site-packages\calibre\library\database2.py", line 3171, in _add_newbook_tag
AttributeError: 'set' object has no attribute 'append'

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! :eek: I would love to have this plugin working.

Lisa

JimmXinu
01-08-2012, 03:46 PM
Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! :eek: I would love to have this plugin working.

@scoopaz, I'm downloading 0.8.34 now--I was still on 0.8.33.

What's the URL of the story that gave the error? Does it happen for all, or just one story?

Jim

scoopaz
01-08-2012, 04:00 PM
Hi Jim,

thanks for the quick response!. The URL for the story I initially tried is :

http://www.fanfiction.net/s/7623520/1/The_Good_Brother

so I tried a few more on fanfiction.net, and got the same error. Went to www.whofic.com just to confirm with a generic (non-adult) story:

http://www.whofic.com/viewstory.php?sid=29654
and received the error again.

Lisa

JimmXinu
01-08-2012, 04:01 PM
@scoopaz, I've figured it out.

It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.

I'll report it in the calibre bug tracking system.

For now, if you remove your "Tags to apply when adding a book", it will work.

Even if you don't want to do that for regular use, please give it a try and confirm that it solves your problem.

Jim

scoopaz
01-08-2012, 04:05 PM
Here's the data from the FFdownloader job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
url:(http://www.fanfiction.net/s/7623520/1/) id:(None)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping


Lisa

scoopaz
01-08-2012, 04:12 PM
Hi Jim,

Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

calibre, version 0.8.34
ERROR: Could not open ebook: File is not a zip file

Traceback (most recent call last):
File "site-packages\calibre\gui2\viewer\main.py", line 64, in run
File "threading.py", line 483, in run
File "site-packages\calibre\ebooks\oeb\iterator.py", line 205, in __enter__
File "site-packages\calibre\customize\conversion.py", line 204, in __call__
File "site-packages\calibre\ebooks\epub\input.py", line 138, in convert
File "site-packages\calibre\utils\zipfile.py", line 751, in __init__
File "site-packages\calibre\utils\zipfile.py", line 786, in _GetContents
File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb :chinscratch:

Lisa

JimmXinu
01-08-2012, 04:21 PM
Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb :chinscratch:
Lisa,

Well, that's not supposed to happen... But on the other hand, it is farther along and therefore progress.

Please try it again making sure that you're adding from the URL and have 'epub' and 'Add New' selected. If it still makes a zero length epub, please post the details from the Download job.

Thanks,

Jim

scoopaz
01-08-2012, 04:32 PM
Hi Jim,

OK, Made sure it was set to add new ebook, and add new from URL and no go.

Here is the results from the job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
url:(http://www.fanfiction.net/s/7623520/1/) id:(None)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping

I tried uninstalling the Goodreads plugin and tried again with no luck. could something be wonky with my configuration?


Lisa

kiwidude
01-08-2012, 04:42 PM
@scoopaz - the two Goodreads plugin messages are due to you having installed two plugins a long time ago that got superceded by the single Goodreads plugin. They have nothing to do with any behavior with the FanFictionDownloader plugin.

It is just calibre telling you that it is not possible to load those plugins (because they arent compatible with recent calibre versions, hence they were replaced with a single "Goodreads" plugin).

To get rid of the error messages, see the couple of posts here (http://www.mobileread.com/forums/showthread.php?p=1825733#post1825733). You can either do it via the command line or as dwnathny says just delete the relevant plugin zip files from the calibre plugins folder. Just make sure you do the right ones - as per your error message text they are "Goodreads Covers" and "Goodreads Metadata" that are the plugins you should remove, and not the "Goodreads" one.

JimmXinu
01-08-2012, 04:56 PM
And I was getting ready to blame it all on Goodreads plugins. :-)

Actually, Lisa, your last log posting gave me the answer.

I've replaced the zip file on the "FanFictionDownLoader 1.1.1" post above with a fixed one. Please give that one a try.

Thanks,

Jim

scoopaz
01-08-2012, 05:06 PM
Hi everybody,

Kiwidude: Thanks for the info - I'll go ahead and get that fixed per your link.

Jim: Uninstalled the old plugin completely and installed the updated one - Success! The story downloaded successfully and opened up without any errors.

:thanks: I suspect I will be abusing this plugin regularly :thumbsup:


Lisa

JimmXinu
01-08-2012, 06:25 PM
Jim: Uninstalled the old plugin completely and installed the updated one - Success! The story downloaded successfully and opened up without any errors.

Excellent! Thanks for helping figure this out.

Jim

JimmXinu
01-09-2012, 07:14 PM
It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.
Turns out this was indeed my fault. I have a fix for it now.

I'll be releasing a new version (1.1.2), so it should be available for auto-update soon.

Jim

JimmXinu
01-10-2012, 09:36 PM
Somebody on the user group requested an option to keep existing tags on updatemeta.

"Existing tags will be kept and any new tags added.
Completed and In-Progress tags will be still be updated, if known.
Last Updated tags will be updated if lastupdate in include_subject_tags."

I also fixed the Last Update tags feature.

LMx
01-12-2012, 05:49 PM
Awesome plugin! thank you very much for your time doing this

my girlfriend is using it often and loves it

however she uses 2 other big websites that are still not supported by this plugin

is there any plans in the future to add support for

http://www.livejournal.com
and/or
http://archiveofourown.org/ ?


also, there were some links that weren't added to the library so i'm trying to help her
here is an example log:

Download FanFiction Book
FanFictionDownLoader v1.1.3
Logfile for book ID None (A Bag Full of Mysteries)
url: (http://www.fanfiction.net/s/6420141/1/) id: (None)
write to c:\users\______\appdata\local\temp\calibre_0.8.34_ tmp__p6hx7\fanfictiondownloader_ayilc3\new-None-dhpynm.epub
Exception: {u'comment': u"'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte", u'publisher': u'www.fanfiction.net', u'good': False, u'pubdate': datetime.datetime(2010, 10, 23, 0, 0), u'author': u'tinybee', u'url': u'http://www.fanfiction.net/s/6420141/1/', u'timestamp': datetime.datetime(2012, 1, 12, 22, 15, 44, 429000), u'author_sort': u'tinybee', u'calibre_id': None, u'is_adult': False, u'comments': u'A series of Harry Potter/Crossover one shots. Pairings will be SLASH Harry/? from numerous crossovers. List of what I can do inside. NO LONGER TAKING PROMPTS.', u'username': 'NoneGiven', u'outfile': u'c:\\users\\______\\appdata\\local\\temp\\calibre _0.8.34_tmp__p6hx7\\fanfictiondownloader_ayilc3\\n ew-None-dhpynm.epub', u'added': False, u'title': u'A Bag Full of Mysteries', u'password': '', u'tags': [u'X-overs', u'Harry Potter', u'In-Progress', u'Romance', 'FanFiction', u'Drama'], u'icon': u'dialog_error.png'}:'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.jobs", line 131, in do_download_for_worker
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.writers.base_writer", line 215, in writeStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 191, in getStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.adapter_fanfictionnet", line 208, in getChapterText
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 167, in _fetchUrl
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte

if mobi format is selected it also gives an error, although a different "FailedToDownload: Error downloading Chapter: http://www.fanfiction.net/s/6420141/24/! Missing required element!"

txt and html formats were added without problems

is it a bug or a fanfiction.net problem? thanks in advance :)

JimmXinu
01-12-2012, 07:18 PM
Awesome plugin! thank you very much for your time doing this

Glad you like it. :)

is there any plans in the future to add support for

http://www.livejournal.com
and/or
http://archiveofourown.org/ ?

livejournal isn't really a fanfic site. From what I remember of it, which admittedly isn't much, it's very free form with little common formatting and story demarcation.

Unless it's changed a lot, it's much more challenging than I, for one, am prepared to tackle for site I don't read myself.

However, if anyone else wants to write an adapter for it, we gladly welcome code contributions.

http://archiveofourown.org has it's own ebook(epub/mobi/etc) downloads. For actual content, they've done a better job than we have--theirs contain images, too. They haven't, however, done as good a job of including metadata for Calibre. I'll think about what I might be able to do there.


http://www.fanfiction.net/s/6420141/24/!
txt and html formats were added without problems

is it a bug or a fanfiction.net problem? thanks in advance :)
This story downloaded fine for me in both epub and mobi. Unless you continue to have problems, I'm going to assume that it was an ffnet glitch that came and went.

Jim

LMx
01-12-2012, 09:15 PM
thank you for the quick reply

indeed, it seems to have been a ffnet glitch as i tried it now again and had no problems

good luck with the improvement of the plugin :)

davidfor
01-12-2012, 09:50 PM
Jim,

I'm not a fanfiction reader, but my wife is, so I had a play with your plugin. I tested it with a couple of the links posted here to see how it went and I am very impressed (and amused by the short Dr Who story).

One thing that occurred to me, do you keep the bookmarks that the Calibre viewer puts into the epub? One of the bookmarks is the current reading location, I can see that if the story is being updated with new chapters, if the bookmarks are lost it could get very annoying. If you haven't seen them, the bookmarks are in the file "calibre_bookmarks.txt" in the "META-INF" folder of the epub.

And I just had a look at the story again. On the title page, you have "Categoryq". Is that a typo?

David

JimmXinu
01-12-2012, 10:08 PM
I'm not a fanfiction reader, but my wife is, so I had a play with your plugin.
Oddly, you're the third guy I've heard from with a similar comment about using it for an SO. My wife just shakes her head tolerantly at me when I talk about fanfic. :)
One thing that occurred to me, do you keep the bookmarks that the Calibre viewer puts into the epub? One of the bookmarks is the current reading location, I can see that if the story is being updated with new chapters, if the bookmarks are lost it could get very annoying. If you haven't seen them, the bookmarks are in the file "calibre_bookmarks.txt" in the "META-INF" folder of the epub.

I did not know it did that. I wonder why Calibre puts it there? It has some other, non-invasive method for mobis...

A quick test appears to show that yes, that file is discarded on epub update.

It shouldn't be terribly difficult to retain it when doing update downloads. Overwrites, of course, will still destroy it.

And I just had a look at the story again. On the title page, you have "Categoryq". Is that a typo?
Ah. Oops.

That's a little embarrassing actually--I changed that on purpose to make sure that the correct file was being used and must have forgot to change it back.

Jim

JimmXinu
01-12-2012, 10:47 PM
It shouldn't be terribly difficult to retain it when doing update downloads. Overwrites, of course, will still destroy it.

Changes to retain calibre_bookmarks.txt through epub updates and fix Categoryq are in 1.1.4. It's uploaded and will start showing up in the automatic plugin list in a while.

Jim

davidfor
01-13-2012, 12:45 AM
Oddly, you're the third guy I've heard from with a similar comment about using it for an SO. My wife just shakes her head tolerantly at me when I talk about fanfic. :)


My wife knows that headshake well. :)


I did not know it did that. I wonder why Calibre puts it there? It has some other, non-invasive method for mobis...


That's interesting. My understanding was that Calibre converted everything to epub to view it. I assumed that it didn't remember the current page if it did this. But, a quick test showed it did.


That's a little embarrassing actually--I changed that on purpose to make sure that the correct file was being used and must have forgot to change it back.

Ah, the deliberate change to help debug something. I know it well. I have had a user see this in my tests and then I spent hours explaining that it wasn't a real change and I was going to remove it.

JimmXinu
01-13-2012, 12:58 AM
That's interesting. My understanding was that Calibre converted everything to epub to view it. I assumed that it didn't remember the current page if it did this. But, a quick test showed it did.
I've never really looked into how the book viewer part works. The full-up conversion process generally takes longer than the viewer starting up, though.

I've just glad you knew about the calibre_bookmarks.txt file. I might never have figured out that problem without it. Seems a pretty Evil thing for Calibre to do, quite honestly. I wonder where it remembers it's place in mobis or other formats?

Jade Aislin
01-13-2012, 12:12 PM
I love this plugin. The only reason I use calibre is to sort the fanfic I read. I’ve barely scratched the surface of the stories I read at FFN and can't wait to try downloading from TTH.

I do have a question however. How do I get the characters to show up? I have been trying to figure out how to get the characters to show up in the title page, but they never do. I’d prefer the characters to show up as tags, but am unsure if it is possible.

I have characters listed in the titlepage_entries section. The FFN story I am using as a test does have two characters, but I do not see it showing up.

One of the urls I am using is: http://www.fanfiction.net/s/7661640/1/

When I tried putting other items like storyID or siteabbrev on the title page I did not have a problem. It is only the characters.

JimmXinu
01-13-2012, 12:31 PM
I do have a question however. How do I get the characters to show up? I have been trying to figure out how to get the characters to show up in the title page, but they never do. I’d prefer the characters to show up as tags, but am unsure if it is possible.
You're doing it right, it sounds like. And when there are characters they will show up as tags by default.

But we only have code for parsing and saving characters for some sites and ffnet isn't one of them. (Currently only FimFiction.net and SIYE do.)

It doesn't look too horrible to add character parsing to ffnet. I'll look into it.

Jim

Jade Aislin
01-13-2012, 01:35 PM
You're doing it right, it sounds like. And when there are characters they will show up as tags by default.

But we only have code for parsing and saving characters for some sites and ffnet isn't one of them.


That's good to know. At least I know I'm not doing something wrong.


It doesn't look too horrible to add character parsing to ffnet. I'll look into it.



I hope you are able to get the characters from FFnet. I've been using a complex method for getting my stories into calibre.

1. use fanfictiondownloader to download story list sorted by character.
2. add stories to calibre.
3. manually input custom fields (character, story complete)
4. repeat steps 1-3 for new character
5. find duplicate stories.
6. Merge duplicate stories so that it shows both characters.
7. Repeat for any additional characters.

This is very time consuming. If you can get the characters in the tags, it would eliminate a lot of the work I have to do.


On another note, is it possible to change the font of the comments? I love that the summary is put into the comments, but have trouble reading it on my tablet. I have tried to bulk edit, but could not make any changes as there are no easy html tags to change. Increasing the font in the comments field while the story is downloaded would be a big help.

JimmXinu
01-13-2012, 01:50 PM
...
This is very time consuming. If you can get the characters in the tags, it would eliminate a lot of the work I have to do.

I'm working on it--but I'm remembering why I didn't before: it's a pain to do correctly. ffnet isn't easy to parse.

On another note, is it possible to change the font of the comments? I love that the summary is put into the comments, but have trouble reading it on my tablet. I have tried to bulk edit, but could not make any changes as there are no easy html tags to change. Increasing the font in the comments field while the story is downloaded would be a big help.

The comments field in calibre's metadata? I'm just shoving plain text into that w/o HTML. calibre's adding a bit of HTML around each paragraph with class="description". I wonder if one could change the def of "description" somewhere.

Jim

JimmXinu
01-13-2012, 04:02 PM
(remove reply to misfiled post)

JimmXinu
01-13-2012, 06:22 PM
@Jade Aislin - I've put up a new version (1.1.5) with parsing and recording of characters on fanfiction.net.

It will be a bit before it shows up in calibre as the latest version.

Jim

Jade Aislin
01-13-2012, 08:14 PM
Wow, you're fast. I tried it and got the characters in the tags. Now I only have to take a few things from the tags and put them into my custom columns. Much easier.

Of course this means I have to sort through all the stories from different fandoms I like on FFN. . .this could take awhile.

But I am definately using your plugin. Thanks.

jackr
01-13-2012, 09:59 PM
First off, amazing plugin :D. It really makes life easier for reading fanfics on my Nook. In fact, it was this plugin that got me to try Calibre.

Anyways, I have quite the list of fanfics now and some of them are flagged as adult material. I wanted to know if its possible to incorporate some sort of 'Yes I am an adult' setting into your plugin so I don't have to sit through the process of checking all the fics for updates and the like.

JimmXinu
01-13-2012, 10:16 PM
@jackr - Glad you like it.

Regarding is_adult: Have you looked at the 'personal.ini' configuration? When you configure the plugin, at the bottom, there's a button to view the defaults for the many settings.

What you want is to add this to your personal.ini settings:

[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
is_adult:true


Looking at it, I see that it's actually not easy to pick that setting out of the defaults file--I'll make it clearer.

You can also set user/pass for different sites that require log in.

Jim

jackr
01-13-2012, 10:39 PM
Thanks for the fast response! I did not look at the personal.ini :o. But it goes to show that if you thought of it, it's probably already covered. :)

This might be the wrong area for it, but is it possible to set up a column to display stats like how many new chapters have been updated, etc?

JimmXinu
01-13-2012, 10:52 PM
Thanks for the fast response! I did not look at the personal.ini :o. But it goes to show that if you thought of it, it's probably already covered. :)
I doubt we've thought of everything, but the FanFictionDownLoader part (as opposed to the plugin part) has been around for a couple years now and I've been working on it for over a year.
This might be the wrong area for it, but is it possible to set up a column to display stats like how many new chapters have been updated, etc?
It can't right now. The closest thing immediately available I know of is the Count Pages (http://www.mobileread.com/forums/showthread.php?t=134000) plugin. But I'm still on a learning curve with advanced calibre use myself.

Setting custom columns from a story's metadata (num chapters, words, etc) is something that could be done in future. I'll add it to the wish list.

Jim

Jade Aislin
01-14-2012, 07:59 AM
I'm loving this now that characters show up in the tags for FFN. It will be nice to use the plugin for updates. I even got the last update to show up in the tags.

I realize that it puts two different formats (year/month and year/month/day), but is there a way to specify which one to use? Or a way to set the format myself, I prefer month/day/year). If there isn't, I won't worry about it. This plugin is still better than any manual input I would have to do otherwise.

cklls
01-14-2012, 09:22 AM
Jim - Loving this plug-in! I'm an avid consumer (and sometimes author) of fanfiction and have happily used your web-based version in the past. Having it built in to Calibre is just... heavenly! If you're open to suggestion on other sites to support, I'd like to request www.dramione.org be added if/when you can. It's a small archive, but rapidly growing in both readers and stories. I'm a staff member there, and would love to be able to use this tool to help with editing and validation.

JimmXinu
01-14-2012, 10:29 AM
@Jade Aislin - You can turn them on and off from personal.ini, but that's it. That feature's actually from one of the other developers--I keep it turned off. :)

@cklss - Which sites get supported depends on the size and activity of the site, how hard it would be to implement, and the interest and free time available to the developer. :)

http://dramione.org is smaller than we usually support, plus right now I'm concentrating on the plugin. So I'm not planning to support it.

However, FFDL (http://code.google.com/p/fanficdownloader/) is an open source project--we welcome other people contributing if you, or anyone, wants to write an adapter to support it.

Jim

cklls
01-14-2012, 11:38 AM
@cklss - Which sites get supported depends on the size and activity of the site, how hard it would be to implement, and the interest and free time available to the developer. :)

http://dramione.org is smaller than we usually support, plus right now I'm concentrating on the plugin. So I'm not planning to support it.

However, FFDL (http://code.google.com/p/fanficdownloader/) is an open source project--we welcome other people contributing if you, or anyone, wants to write an adapter to support it.

Jim

Thanks, Jim. Not at all surprised that it's too small for your attention. :) I'll mention it to our tech person to see if she's got time to fiddle with FFDL. Apart from clicking on the link, I honestly wouldn't know where to begin! The last "code" I wrote was probably close to thirty years ago, "programming" a bunny to print from DOS commands. HTML emphasis codes (italics, bold, etc.) are about as far as I go! LOL

JimmXinu
01-14-2012, 12:41 PM
IMO, it's not hugely difficult. I'd suggest looking at the adapter for castlefans.org (http://code.google.com/p/fanficdownloader/source/browse/fanficdownloader/adapters/adapter_castlefansorg.py) as an example.

Jim

Jade Aislin
01-14-2012, 07:04 PM
@Jade Aislin - You can turn them on and off from personal.ini, but that's it. That feature's actually from one of the other developers--I keep it turned off. :)


That's alright. I'm just happy it's there at all. I can work with what it gives me.

However, FFDL (http://code.google.com/p/fanficdownloader/) is an open source project--we welcome other people contributing if you, or anyone, wants to write an adapter to support it.

Jim
Hmm. . .I might try to get some other sites I visit adapted, http://www.masterapprentice.org/ (a star wars site) and http://dokuga.com/fanfiction or http://dokuga.com/spark (two inuyasha sites). All of them have between 4,000 to 5,000 stories. Unfortunately, it has been over ten years since I did any programing and I had decided then that I wasn't very good at it. But I'll give that link you gave cklss a look.

JimmXinu
01-15-2012, 01:19 PM
@Jade Aislin - You'd asked about changing the appearance of the comments field:

The comments field in calibre's metadata? I'm just shoving plain text into that w/o HTML. calibre's adding a bit of HTML around each paragraph with class="description". I wonder if one could change the def of "description" somewhere.

I happened to find where this is set while looking for something else.

The general answer is in Customizing calibre, Overriding icons, templates, etcetera (http://manual.calibre-ebook.com/customize.html#overriding-icons-templates-etcetera).

First, find your calibre config dir by doing Prefs->Misc->Open calibre conf dir. Go into resources, make a directory templates.

Then, copy C:\Program Files (x86)\Calibre2\resources\templates\book_details.cs s (or equivalent) to the templates dir you just made.

Edit your new book_details.css file and add:
.description {
font-size: 25px;
}

...or whatever size you like.

I believe that will do it for you.

Jim

Jade Aislin
01-15-2012, 03:06 PM
That worked perfectly! Once I figured out where to place it. Now I don't have to go through thousands upon thousands of epubs to individually edit their summaries. I've been trying to figure out what to do for months.

Thank you.:2thumbsup:thanks:

JimmXinu
01-16-2012, 03:37 PM
I'm again looking for some people to beta test the attached newest version before I put it out for everyone.

This version contains:

New feature - Options to update Reading List plugin lists for device sync and keeping list(s) of new books to read.
Plugin configuration now tied to library. Different libraries can have different settings.
Shortcut button in plugin config to dialog for assigning hotkeys.
List of Characters now parsed and saved for most supported sites.


The big change is the integration with Reading List (http://www.mobileread.com/forums/showthread.php?t=134856).

Reading List defines lists of stories that it can automatically send to your reading devices and update tags and custom columns.

Once you've created Reading Lists for sending to your devices, you can configure FanFictionDownLoader 1.2.0 to automatically add new/updated stories to those lists.

Reading Lists can also update custom columns automatically. Using that with a Metadata Plugboard, you can put a notation in the title of new books to read when they're sent to your device.

As a convenience, the plugin offers a menu option to remove stories from the 'to read' lists after you've read them and optionally add 'read' stories to the 'send to device' lists again.

This is an illustrated example (http://code.google.com/p/fanficdownloader/wiki/FanFictionDownLoaderPluginWithReadingList) of how I thought it would be used.

There's also a decent chance somebody will point out a better way to do this. :)

Thanks,
Jim

soleta
01-18-2012, 11:33 PM
Thank you so much for this plugin - it's honestly one of the ones I use most often. Because of that, I was wondering if there's a way I'm missing to tell the plugin not to add tags to the file, or to use calibre's default 'tags to apply when adding a book' setting?

todaysname
01-19-2012, 06:59 AM
Wonderful plugin, I used it all the time up until the latest update. Past updates worked well.

Now every time I open the plugin to either add a new url or update a new story, Calibre has to quit unexpectedly. I'm not a technical person at all. If there is some data that you need about the crash, let me know where to look and I will post that information.

JimmXinu
01-19-2012, 11:51 AM
Thank you so much for this plugin - it's honestly one of the ones I use most often. Because of that, I was wondering if there's a way I'm missing to tell the plugin not to add tags to the file, or to use calibre's default 'tags to apply when adding a book' setting?
Glad you like it.

There's meant to be a way, but when I went to try it, I found a couple bugs that I need to fix before it will work properly. Stay tuned.

Jim

JimmXinu
01-19-2012, 03:21 PM
Version 1.2.2 is up. After a while, calibre will notice it and prompt you to update.

@soleta - Once you have 1.2.2, you can go into plugin config->personal.ini and add a line:
[defaults]
include_subject_tags:

(Don't add another [defaults] line if you already have one.)

That should keep the plugin from writing any tags to calibre. I'd turn on 'Keep Existing Tags' on the 'Basic' config tab, too.

Jim

soleta
01-20-2012, 10:43 AM
Thank you, thank you! That's perfect. :D

richreads
01-22-2012, 10:52 PM
Hi.
Love the plugin.
I just started using it to add stories from fanfiction.com.
It does add the story but then I get an error when I sort by date.
If I shut down calibre and restart it i can sort by date without the error reappearing.
The error also disappears (with restarting calibre) if I remove the story that was just added.
I've added a book manually with out using the plugin and did not get the error so it seems to be the plugin.
I tried with two different authors and received a message both times.

Here is the error message.
calibre, version 0.8.36
ERROR: Unhandled exception: <b>TypeError</b>:can't compare offset-naive and offset-aware datetimes

Traceback (most recent call last):
File "site-packages\calibre\gui2\library\models.py", line 319, in sort
File "site-packages\calibre\gui2\library\models.py", line 327, in _sort
File "site-packages\calibre\library\caches.py", line 995, in sort
File "site-packages\calibre\library\caches.py", line 1015, in multisort
File "site-packages\calibre\library\caches.py", line 1031, in __cmp__
TypeError: can't compare offset-naive and offset-aware datetimes

JimmXinu
01-22-2012, 11:09 PM
Hi.
Love the plugin.
I just started using it to add stories from fanfiction.com.
It does add the story but then I get an error when I sort by date.

Glad you like it. :)

As for the bug--it's fixed in 1.2.3. Thanks for reporting it.

Jim

JimmXinu
01-22-2012, 11:15 PM
FYI, anytime there's a new version of a plugin, there's a delay before it shows up on the official list of plugins that calibre watches. (It's managed manually by kiwidude and Kovid.)

But you don't have to wait for it. You can go into your User Plugins, view your Installed plugins, choose FanFictionDownLoader and, even though it says there's no new version, click 'Install'. Restart calibre and you'll have the latest version.

Jim

richreads
01-23-2012, 11:37 AM
thanx for the quick update - that fixed it.

martinjh99
01-24-2012, 11:41 PM
Latest version of the plugin and Calibre 0.8.36 Windows 7 64bit...

Download and install the plugin from both the plugin list in Calibre and manually downloaded and installed from the first post in this thread.

It doesn't show up in the Toolbar options so i can add it and when i click on the plugin to configure it it always says that I need to restart Calibre when I have already re-started Calibre.

Am I doing anything wrong here or is it a problem with soomething...?

theducks
01-25-2012, 11:14 AM
Latest version of the plugin and Calibre 0.8.36 Windows 7 64bit...

Download and install the plugin from both the plugin list in Calibre and manually downloaded and installed from the first post in this thread.

It doesn't show up in the Toolbar options so i can add it and when i click on the plugin to configure it it always says that I need to restart Calibre when I have already re-started Calibre.

Am I doing anything wrong here or is it a problem with soomething...?

You either missed the configuration step, or you placed it on the wrong (for you) toolbar. There are 2 versions of the same toolbar: Normal and Device connected. If you want it on Both, you need to add it to both.

Spending some time with Preferences Toolbar will set things straight. :thumbsup:

martinjh99
01-25-2012, 11:20 AM
You either missed the configuration step, or you placed it on the wrong (for you) toolbar. There are 2 versions of the same toolbar: Normal and Device connected. If you want it on Both, you need to add it to both.

Spending some time with Preferences Toolbar will set things straight. :thumbsup:

Thanks - actually seems to have worked now I am in a library with books I downloaded from website...

Cheers anyway!

JimmXinu
01-25-2012, 11:28 AM
Hmmm. Not sure what happened there; what you have in your library shouldn't make any difference. But I'm glad it got sorted out.

martinjh99
01-25-2012, 12:55 PM
Yeah thanks Jimmy nice little plugin - Was looking for something to make it easier to read longer stories from Fanfiction.net :D

Cheers Mate

JimmXinu
01-27-2012, 05:10 PM
I'm again looking for some people to beta test the attached newest version before I put it out for everyone.

This version contains:

New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.
New feature: Collect and update Series column.
Improved drag-and-drop of URLs onto the Add New URL text box. Dragging URLs in adds now instead of replaces, and copies the URL, not the pretty text.
Added a 'show this again' checkbox to the dialog for reviewing updates before applying them. When unchecked, updates will be applied immediately if there were no problems.
Added a progress bar while applying updates, which can be lengthy.
Don't disable menu items(like update) when there are no books selected. It broke the keyboard shortcut if the menu hadn't rebuilt since books were selected.
Make the FanFictionDownLoader button (which does add or update, depending on whether books are selected) available for keyboard shortcut.
Fix date change for adastra.
Fix URL pattern, metadata collection for SIYE.

I appreciate your feedback.

Adding Series support turned out to be easier than I expected and there's not been a lot of testing on this yet, so I rolled it into 1.3.1.

(Look at the personal.ini config tab if you don't want the series column to update, but you do want other calibre Metadata updated.)

Thanks,
Jim

JimmXinu
01-27-2012, 11:56 PM
This version contains:

New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.
New feature: Collect and update Series column.



Second new feature added to the most recent beta now. FYI.

x85849
01-28-2012, 03:36 PM
New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.

Thank you for this feature. I just started using this plugin and was trying to figure a way to fill the rating info into a custom column in calibre. Lo and behold you updated the plugin with just that feature. Thank you.

The copy to custom column feature seems to work well with the stories I pulled from fanfiction.net. At first it did not appear to work with the rating setting, however I realized that the issue was that my custom column was using a "text, but with a fixed set of permitted values" field and I did not have all of the story ratings in my list. I created a new column of just text and it imported the rating just fine after that.

JimmXinu
01-28-2012, 03:39 PM
@x85849 - You're quite welcome. :)

Jade Aislin
01-28-2012, 06:06 PM
And here I thought I'd have to fill out my custom columns manually from the data in the tags. I love this new feature. The series feature is nice too, at least for those sites that have series. It's too bad that FFN doesn't. Still, I am very pleased with this plugin.

Good work!

jackr
01-30-2012, 01:09 AM
Amazing work! Solved a number of things on my to do list :D.

I don't know if the actual site adapter is the problem, but fimfiction.net doesn't split "My Little Pony" off into Category and the genre tags off into the Genre section.

And pardon me if this is a noob question :o, but can the progress windows be placed inside the jobs section instead? I have a large list of fanfics now that I regularly check for updates, and I can't go off and do something else in Calibre until the metadata downloading is finished.

JimmXinu
01-30-2012, 12:56 PM
Amazing work! Solved a number of things on my to do list :D.
Glad you like it. :)

I don't know if the actual site adapter is the problem, but fimfiction.net doesn't split "My Little Pony" off into Category and the genre tags off into the Genre section.
I didn't write the fimfiction.net adapter myself, but that should be easy to fix.
And pardon me if this is a noob question :o, but can the progress windows be placed inside the jobs section instead? I have a large list of fanfics now that I regularly check for updates, and I can't go off and do something else in Calibre until the metadata downloading is finished.
Metadata download runs in the foreground so it can stop and ask for user/pass if needed and check the existing book vs the site. I'll think about it, but I don't see that changing.

IMO, you should use the email notifications from the sites to inform you when your stories are updated. Using this to poll them all everyday seems rather wasteful to me and inflates the stories' hit counts.

Jim

JimmXinu
01-30-2012, 09:45 PM
Version 1.3.2 - 30 Jan 2012
New site: archiveofourown.org -- Thanks for Ida Leter for implementing this.
Change fimfiction.net to use genre instead of category.

jackr
01-30-2012, 11:57 PM
Thank-you for the fix :D. I wish I found this sooner :o, but I found another issue. I am not sure what is going on, but for my fimfiction.net fics that had updates in Jan, the year is set at 2011 instead of 2012. Not sure if the fimfiction forgot to roll over the year or not :P.

JimmXinu
01-31-2012, 03:52 PM
I am not sure what is going on, but for my fimfiction.net fics that had updates in Jan, the year is set at 2011 instead of 2012. Not sure if the fimfiction forgot to roll over the year or not :P.
Oh, it's worse than that. fimfiction doesn't give the year anywhere so we left it hardcoded to 2011. We can at least do better than that.

Somebody might want to suggest to the fimfic people that they add a year before Jun, when their oldest story is dated.

JimmXinu
01-31-2012, 04:00 PM
@jackr - There's a new version, 1.3.3, that handles the fimfic dates better, but not perfectly. Now it assumes "this year", unless that date would be in the future, then it subtracts a year.

Until fimfiction.net adds the year, that's as good as it's going to get.

jackr
02-01-2012, 05:46 AM
Thanks for the update! I sent a request to the creator of fimfiction asking if a year could be added.

leistnerm
02-02-2012, 12:14 AM
Any chance of getting any of the following sites supported?

http://www.fanficauthors.net/
http://www.hpfanficarchive.com/
http://thequidditchpitch.org/
http://www.phoenixsong.net/
http://restrictedsection.org/
http://www.mugglenet.com/
http://checkmated.com/
http://mvsfiction.com/
http://sugarquill.net/

... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.

JimmXinu
02-02-2012, 12:32 PM
Any chance of getting any of the following sites supported?
...
It's unlikely that I'm going to write adapters for them. But we're happy to accept other people's code to support new sites.
... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.
There's been a recent discussion about "Writing adapters" on the Google group:

http://groups.google.com/group/fanfic-downloader/t/7d3091e8dd61e961

I suggest you start there.

Jim

leistnerm
02-02-2012, 10:20 PM
Jim -- Thanks for the link. That definetly helps point me in the correct direction. I've been looking over the code to get an idea of how it's structured and how python works. So far it seems pretty straight forward.

I did notice one thing that I'll probably work on, but just in case I can't figure it out, I'll let you know too ;) It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.

I did manage to find a few pages on how to do this in python, but not with the method you are using to retireve urls. So I'll probably play around with that for a starter... something easy before I try to do a whole file myself.

http://stackoverflow.com/questions/3947120/does-python-urllib2-will-automaticly-uncompress-gzip-data-from-fetch-webpage
http://techknack.net/python-urllib2-handlers/

Anyways, thought I'd at least metion my previous experience with a progam doing something similar to this and the speed increase I got.

Looking forward to hopefully being about to eventually contribute to this already great plugin.

Mark

JimmXinu
02-02-2012, 10:57 PM
It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.
That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

But I don't remember ever thinking about it on our client side. It's certainly worth a look, thanks for the suggestion.

BTW, ff.net in particular is slower than it could be--there's a hard coded 0.5sec sleep between hits (plus the slow_down_sleep_time, if set). We have first hand evidence from one of the other developers that ff.net will restrict your IP if you hit them "too fast". See http://www.fanfiction.net/tos/ section 4E.

Jim

leistnerm
02-02-2012, 11:17 PM
That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

Jim

Hmm. I've been pegging them pretty hard for a while and not been blocked. I guess I should be careful on that. And after further digging, it looks like it has code to support gzip, in the __decode method/function/whatever python calls them. Perhaps the slowdown is just due to that delay.

Anyways thanks for your time :)

JimmXinu
02-03-2012, 12:35 AM
Using the GZipProcessor from http://techknack.net/python-urllib2-handlers/, it's actually really easy to implement.

Some fairly cursory testing seems to indicate that there's no real improvement for ff.net. And several of our supported sites won't send us gzip (looks like it's due to User-Agent). I did see ~30% improvement with TtH, though.

That's as much effort as I'm going to put into it tonight. :)

ilovejedd
02-03-2012, 10:53 AM
Just tried it last night. Excellent plugin. Wish it was available 3 years ago when I started converting fanfics to ebooks.

iatheia
02-05-2012, 10:38 PM
Any chance of getting any of the following sites supported?

http://www.fanficauthors.net/
http://www.hpfanficarchive.com/
http://thequidditchpitch.org/
http://www.phoenixsong.net/
http://restrictedsection.org/
http://www.mugglenet.com/
http://checkmated.com/
http://mvsfiction.com/
http://sugarquill.net/

... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.

I can take a look at a few more populated ones in a bit.

Ida.

Ayiana
02-05-2012, 10:59 PM
First off, thank you so much for crafting this plugin. In the past I've spent untold hours downloading fics manually, adding them to Calibre, and then filling in all the fields. This plugin, if I can get it to work right, is the answer to a prayer.

Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing? If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 241, in do_loop
IndexError: list index out of range

I've tried uninstalling and reinstalling the plugin several times, even with reboots/restarts in between, to no avail. Any ideas?

JimmXinu
02-06-2012, 10:17 AM
Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing?
I don't think I've ever seen the button disappear after it was on screen--and believe me, I crashed it a lot during development. :)
If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 241, in do_loop
IndexError: list index out of range
The only way I know to get that error is to click the plug in with no books selected (or choose 'Add New from URL(s)'), then *not* enter any URLs and click OK.

If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful.

Jim

Ayiana
02-06-2012, 10:53 AM
Well, I'm utterly baffled now.

I uninstalled the plugin and shut down my computer last night right after I posted here. Reinstalled this morning, set my preferences, and ... everything's fine. No crash. No error message. Color me confused but relieved.

And thank you again. This plugin is a fantastic addition to Calibre's functionality.

JimmXinu
02-06-2012, 11:15 PM
@Ayiana - Glad you like it. Hopefully it stays fixed.

Laia
02-07-2012, 01:13 PM
Hi Jim,
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Thank you

davidfor
02-07-2012, 08:45 PM
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Save the file to disk. Then open Calibre and it's preferences. Click on "Plugins" (near the bottom-left of the window) and then click on "Load plugin from file". In the file picker window, find the pligin zip file, select it and press "Open". Then follow the rest of the prompts to install and configure.

JimmXinu
02-07-2012, 08:50 PM
@Laia - What davidfor said is correct.

Or, you don't need to download it yourself--calibre can do it for you.

For full instructions on installing plugins, see:
http://www.mobileread.com/forums/showthread.php?t=118680

jjm6383
02-07-2012, 09:23 PM
Hi,

I really like this plugin. It's making my life easier.

I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?

Thanks,
Joyslin

JimmXinu
02-08-2012, 06:13 PM
I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?
Glad you like it.

Do you use the HTML output? epub, mobi and obviously txt are more limited in what they can use for CSS. And the story text HTML is taken from the source sites as untouched as possible. Deffnetizer only deals with ffnet, which for all its sins is extremely consistent in it's HTML. Trying to wrap custom html and CSS around all the different sites would be... interesting, I fear.

I'll add user CSS to the wish list, but my free time has largely dried up again for a while, so don't hold your breath. :)

anesthezea
02-08-2012, 09:47 PM
On some stories (not all) I get the following error:

'NoneType' object has no attribute 'parent'

Show Details:

calibre, version 0.8.37
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 241, in do_loop
IndexError: list index out of range

Help?

JimmXinu
02-08-2012, 09:55 PM
You're the second person to report an error occurring on that line.

The only way I know to get that error is to click the plug in with no books selected (or choose 'Add New from URL(s)'), then *not* enter any URLs and click OK.

(Which is a case I should start checking for, but haven't yet.)

If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful.

Or if it happens consistently on the same stories, knowing the URLs for those stories would help me a lot.

anesthezea
02-09-2012, 12:17 AM
ETA: I updated the plugin (didn't realize it was one version out of date) and the "parent" error seems to have been fixed. Thank you!

JimmXinu
02-11-2012, 03:05 PM
Testing on this version has been finished.

I'm again looking for some people to beta test the attached newest version before I put it out for everyone.

This version contains:

New feature - Allow user to set custom CSS in personal.ini for HTML and EPUB output.
New feature - Allow user to set custom regular expressions in personal.ini to modify metadata.
New feature - Use Accept-Encoding=gzip to speed download. (Not sites will use it--it's common for sites to block gzip based on User-Agent.)
Add progress bars while collecting URLs from stories for list and for updates.


Here's an example of what the default output_css parameter for epub looks like:
## output background color--only used by html and epub (and ignored in
## epub by many readers). Included below in output_css--will be
## ignored if not in output_css.
background_color: ffffff

## Allow customization of CSS. Make sure to keep at least one space
## at the start of each line and to escape % to %%. Also need
## background_color to be in the same section, if included in CSS.
output_css:
body { background-color: #%(background_color)s;
text-align: justify;
margin: 2%%; }
pre { font-size: x-small; }
sml { font-size: small; }
h1 { text-align: center; }
h2 { text-align: center; }
h3 { text-align: center; }
h4 { text-align: center; }
h5 { text-align: center; }
h6 { text-align: center; }
.CI {
text-align:center;
margin-top:0px;
margin-bottom:0px;
padding:0px;
}
.center {text-align: center;}
.cover {text-align: center;}
.full {width: 100%%; }
.quarter {width: 25%%; }
.smcap {font-variant: small-caps;}
.u {text-decoration: underline;}
.bold {font-weight: bold;}

Here's an example of what some replace_metadata lines might look like.

## Use regular expressions to find and replace (or remove) metadata.
## For example, you could change Sci-Fi=>SF, remove *-Centered tags,
## etc. See http://docs.python.org/library/re.html (look for re.sub)
## for regexp details.
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
replace_metadata:
(.*)-Centered=>
Magical Girl Lyrical Nanoha=>Nanoha
Puella Magi Madoka Magica.*=>Madoka
(Friend)(ship)=>\2\1
(.*)Great(.*)=>\1Moderate\2
(?s)(.*)suck(.*)=>\1kcus\2

"(.*)-Centered=>" removes all Name-Centered tags entirely (TTH).
"Puella Magi Madoka Magica.* =>Madoka" changes ffnet's "Puella Magi Madoka Magica/魔法少女まどか★マギカ" to just Madoka.
"(Friend)(ship)=> \2\1" changes Friendship to shipFriend. Not useful, but demonstrates regular expression group substitution.
"(.*)Great(.*)=>\1Moderate\2" shows changing one word in the middle of a tag, such as a series name.
"(?s)(.*)suck(.*)=>\1kcus\2" (?s) tells the regexp engine to allow . to make newlines, too. Needed to affect many descriptions/summaries.

The replacement pattern must match the entire string for a piece of metadata to be changed. "Friend" will not match category "Friendship", but "F.*" will. Replacements are applied to all metadata, so be careful to be specific.

You can have different lists of replacements for different sites. One use could be to normalize the rating tags different sites use to all be the same.

JaneD
02-13-2012, 07:51 PM
I love this plugin. Such a convenience to be able to load stories directly into calibre!

My only problem with it is that I seem to be getting some visible code on italics, elipses or dashes, and sometimes at the end of sentences. For example, on this story on AO3: http://archiveofourown.org/works/338120/chapters/547213.

Thank you so much for this wonderful plugin! I particularly like how it will grab an update on the latest chapter of a story without having to re-enter the story url. Brilliant. :thumbsup:

JimmXinu
02-13-2012, 08:39 PM
@JaneD - Glad you like it.

Those "&#xA0;" etc entities showing up are from a bug in the downloader--thanks for pointing it out.

I'll have a fixed version up later this evening.

JaneD
02-14-2012, 10:58 AM
Just reloaded the story and it works perfectly. Thank you for the quick reply and update!

Jade Aislin
02-14-2012, 02:55 PM
I love the feature that puts tags into custom columns, especially with FFN stories. However, I have recently noticed that not all of the characters are getting put into the correct column. Most of the stories will put the characters in, but any story that has Harry Potter as a category does not grab any characters. The characters do not get put into my tags either. I'm not sure if there is any other category having this problem.

Here is one of the urls that should have characters: http://www.fanfiction.net/s/5930194/1/

Edit: Looking on my list of stories, I realized that all the stories that are not retrieving the characters are crossovers. Each story has to categories or fandoms. I do not know why the plugin is having trouble retrieving the character information with these stories, but it is not just stories in the Harry Potter category.

JimmXinu
02-14-2012, 09:29 PM
@Jade Aislin--The hoops I have to jump through to parse out characters on ffnet are significant. It doesn't surprise me that it was flawed.

But I think I have it figured out. Fix will be in the next version.

Jade Aislin
02-14-2012, 11:42 PM
@Jade Aislin--The hoops I have to jump through to parse out characters on ffnet are significant. It doesn't surprise me that it was flawed.

But I think I have it figured out. Fix will be in the next version.

That's great. I really appreciate this feature and the effort you went through to make it as I don't need to pre-sort the stories by character.

Truthfully, I may not have noticed it if I hadn't been thinking about using the new feature that allows the user to edit metadata. I already tried it by modifying some long categories and some categories that have commas in them which made the pluging think there were two categories instead of one (ex: House, M.D.). It worked great, but I was thinking about using the feature to change the character based on the category so that it reads as category.character instead of just the character. Even if it can't be done, it's an easy matter to manually change the characters if I already have the character and categories.

JimmXinu
02-15-2012, 10:05 AM
... but I was thinking about using the feature to change the character based on the category so that it reads as category.character instead of just the character. Even if it can't be done, it's an easy matter to manually change the characters if I already have the character and categories.
Crossover stories can have multiple categories, making it prohibitively difficult to do in the downloader.

So I think you're on your own for that one. :)

85eren
02-19-2012, 06:42 AM
I love this plugin. Thank you.

I don't know if I'm doing something wrong, but none of the files I've downloaded using the plugin, have chapter breaks on the Locations bar on the bottom of the page, when I put them on the kindle.
What am I doing wrong?

iatheia
02-19-2012, 09:06 AM
I love this plugin. Thank you.

I don't know if I'm doing something wrong, but none of the files I've downloaded using the plugin, have chapter breaks on the Locations bar on the bottom of the page, when I put them on the kindle.
What am I doing wrong?

That is because mobi as a proprietary format needs to die in fire. Advise is to use epub, and then convert it to mobi using Calibre, then it sort of looks normal.

JimmXinu
02-19-2012, 10:30 AM
That is because mobi as a proprietary format needs to die in fire.
:thumbsup: Exactly!

I spent the better part of a week once trying to get those chapter marks to work in our mobi output and failed. I had removed mobi output support entirely at one point (before the plugin), but the web site users asked for it back, even if it is flawed.

JaneD
02-19-2012, 01:50 PM
That is because mobi as a proprietary format needs to die in fire. Advise is to use epub, and then convert it to mobi using Calibre, then it sort of looks normal.

This is what I do, and it looks perfect on my kindle, chapter breaks, TOC & all. Just remember to convert again every time you update a chapter! :thumbsup:

JimmXinu
02-19-2012, 01:57 PM
This is what I do, and it looks perfect on my kindle, chapter breaks, TOC & all. Just remember to convert again every time you update a chapter! :thumbsup:
I highly recommend the Reading List plugin for sending books to kindle. Using Reading List, calibre automatically converts to mobi and sends the files to my kindle when I plug it in.

The FFDL plugin is even integrated with Reading List (http://code.google.com/p/fanficdownloader/wiki/FanFictionDownLoaderPluginWithReadingList).

85eren
02-20-2012, 03:11 PM
Thank you for all your help. Getting the file in epub and converting to mobi worked great.

jackr
02-20-2012, 05:18 PM
Is it possible to grab an image (if the site has images for their fanfictions) and apply it automatically to the cover? And if there is no cover, an option to have Calibre generate a cover?

I can survive by copy-pasting the images, but if fanfic images get popular on other sites, it would be a nice feature to have :D.

JimmXinu
02-20-2012, 06:17 PM
@jackr - I'd like someday to d/l and include images, but it doesn't now. Which site(s) are you talking about having cover images?

Have you looked at the Generate Cover (http://www.mobileread.com/forums/showthread.php?t=124219) plugin?

jackr
02-20-2012, 10:43 PM
Namely fimfiction.net. A good portion of the fics there have cover art for them. Ex: http://www.fimfiction.net/story/8933/ and http://www.fimfiction.net/story/2766/.

I have played with Generate Cover a little bit, makes adding the word 'Complete' in the center of every cover for completed fics a snap for Nook browsing. :D

Your new EpubMerge plugin looks great too; I will need to remind myself to only use it on the device however. :)

mhomann
02-21-2012, 06:24 AM
I have come across a fanfic that gives me a book with 40 empty chapters in calibre:
http://www.fictionalley.org/authors/barb/replay.html

what do i do about that?

JimmXinu
02-21-2012, 03:42 PM
I have come across a fanfic that gives me a book with 40 empty chapters in calibre:
http://www.fictionalley.org/authors/barb/replay.html

what do i do about that?
Reporting it here is good. New version with a fix for that is coming shortly.

cklls
02-22-2012, 12:13 PM
Hi Jim - Just tried to download a completed story from ff.net and got this error message:

coercing to Unicode: need string or buffer, NoneType found

The url of the story is:

http://www.fanfiction.net/s/6141793/1/In_Perfect_Light

Any ideas?

JimmXinu
02-22-2012, 12:15 PM
ffnet changed their format slightly--just enough to break things, really.

Update to the latest version of the plugin and it should be good again.

cklls
02-22-2012, 12:21 PM
They do that often, the buggers. ;) I got the push update a couple of hours ago. Is there another one?

JimmXinu
02-22-2012, 12:22 PM
@cklls - v1.4.5 fixes the problem for me. If it doesn't fix it for you, please let me know and I'll look at it again.

cklls
02-22-2012, 12:32 PM
Just installed that version and tried again. (I had 1.4.4 this morning.) No go - same error.

EDIT: Never mind... I forgot to restart Caliber again. With restart, it works. Thanks!!!

JimmXinu
02-22-2012, 12:36 PM
Oh good--you had me worried for a moment. :)

cklls
02-22-2012, 12:39 PM
:thanks: Don't worry, be happy! It's all good. :)

paegan
02-22-2012, 09:07 PM
Ok first off kudos to you on such a KICK @$$ plugin!:thumbsup: The only other plugin I've found almost as useful is the aforementioned Cover Generator (it's fun to download fonts matching the series to make a halfway decent lookin book cover)

I was a little confused for a bit because, while looking up references to your plugin, I hit upon this site : http://www.fanfictiondownloader.net/ and thought you had made this program and couldn't understand why the calibre plugin didn't have as much features.

I realise now that's not the case but I don't know if you want to get into contact with that author to see if he/they are willing to share info on how they do things. For now I only use it to grab fics from sites you don't support (mostly pervy/adult oriented fic sites:p) but it leaves a lot to be desired in regards to metadata crawling and toc generation. Yours is FAR superior in those areas and also includes the story update function which is a godsend.

My only possible suggestion would be to add support for more sites even without full functionality i.e. toc or metadata. I personally would not mind basic support for the oddball site at least just to grab a fic in its entirety without having to depend on an autopager or worse CTRL+C/V.

Keep up the good work and hope to see lots more in the future...

paegan
02-22-2012, 09:15 PM
Almost forgot...
Dunno if this falls within the scope of your plugin but as an additional feature for the book id's if it would be possible to use customised ids to reference fanfic sites e.g.

http://www.fanfiction.net/s/5149431/1/Stranger_Than_Fanfiction
would be something like:
ffn:5149431

or http://ficwad.com/story/181839
would be fwad:181839

Hopefully you get the idea I'm tryin to put accross

JimmXinu
02-22-2012, 09:19 PM
http://www.fanfiction.net/s/5149431/1/Stranger_Than_Fanfiction
would be something like:
ffn:5149431

or http://ficwad.com/story/181839
would be fwad:181839

Hopefully you get the idea I'm tryin to put accross
Meaning you want to see those somewhere? Because siteabbrev and storyId are already a couple of the metadata fields the plugin makes available.

If you're asking to be able to enter those instead of URLs, I suppose it's possible, but I don't see the point--it's a lot easier, IMO, to copy or drag the URL.

paegan
02-22-2012, 09:52 PM
Meaning you want to see those somewhere? Because siteabbrev and storyId are already a couple of the metadata fields the plugin makes available.

If you're asking to be able to enter those instead of URLs, I suppose it's possible, but I don't see the point--it's a lot easier, IMO, to copy or drag the URL.

Hmmm tried it... dosent work exactly like I hoped it would but meh, it's wasn't a necessity just a nice bit of polish (although if some1 else writes a plugin that does do that I do hope that you'd support it :D)

And to toss another feature idea @ you (which kinda obliquely relates to the custom ids thing):

Story mirrors i.e. the same story from multiple sources eg.
http://ficwad.com/story/182760 and
http://www.fanfiction.net/s/7820904/1/The_bLife_b_bBond_b

Now feel free to tell me where to get off cuz I kno this would be insanely difficult to implement eg. having to deal with choosing which link to update from, which like to crawl for metadata, etc. etc.

Nothing ventured, nothing gained ...

JimmXinu
02-22-2012, 10:03 PM
Hmmm tried it... dosent work exactly like I hoped it would but meh, it's wasn't a necessity just a nice bit of polish (although if some1 else writes a plugin that does do that I do hope that you'd support it :D)
I still don't understand what it is you want here. If you want 'ffnet:1324' to appear in calibre, put siteabbrev and storyId each in a custom column and use a third composite column to display them together. I use that to have a wordcount/chapters column, like: 12,123/7.
Story mirrors i.e. the same story from multiple sources eg.
http://ficwad.com/story/182760 and
http://www.fanfiction.net/s/7820904/1/The_bLife_b_bBond_b
I don't see that there's anything particularly useful to be done there. I've seen the same story on different sites with different names, or chaptered differently, or different author name, etc... You can't make any useful assumptions.

JimmXinu
02-24-2012, 05:47 PM
New Version up with support for a new site: gayauthors.org. Thanks to Ida Leter for implementing this.

JimmXinu
02-27-2012, 02:18 PM
Testing for 1.5 is finished. It's been released for general use.

These changes are all for EPUB format ebooks.

The big changes are:

Support for downloading images in the story text.
Support for cover image from three different sources.
Support to resize, make grayscale, and convert images to jpg.
Option for plugin to update calibre cover image on metadata update.
Major re-write of EPUB update to support images.
Improved handling of ini section priority.
New 'Stay on Top' option for 'Add New from URL(s)' dialog for easier drag-n-drop.
Add replace_hr option to replace <hr> tags with '* * *'
Support for fimfiction.net's locked story passwords.


Images
There's several new config parameters you can add to your personal.ini to turn on these features.

All of them should go under the [epub] section or the epub section for individual sites, such as [www.ficwad.com:epub].

The first turns images on in general. Without this, no images will be collected and it defaults to false.
include_images:true

The next allows the summary section in the title page to preserve its HTML from the source site. That way, images in the summary will be included. More on that in a moment. Summary metadata given to calibre still has HTML stripped for consistency.
keep_summary_html:true

As a convenience, and the draw the attention of more casual users, I added an 'Include images in EPUBs' checkbox to the Basic config tab. This is equivalent to adding:
[defaults]
include_images:true
keep_summary_html:true
make_firstimage_cover:true
...to the top of personal.ini. Your settings in personal.ini will override this.

Covers
There are three places cover images can come from.

First is an explicit cover image from the source site. Today, fimfiction.net is the only one--please let me know (with example URLs) if any of the other sites do, too.

Second, the downloader can use the first image it finds in the story with this paramter. If keep_summary_html is also true, it can come from the summary.
make_firstimage_cover:true

Third, you can set a specific image that will be used if one of the other two isn't found.
[www.adastrafanfic.com:epub]
default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
[www.fictionalley.org:epub]
default_cover_image:http://www.somesite.com/someimage.gif

Finally, if you want images inside the EPUB, but never want an image cover, you can use:
never_make_cover:true

Image options
All images saved to EPUB are converted to jpeg to save size. Further, you can set what size to scale the images down to fit in. Aspect ratio is preserved and images are never scaled up, only down. You can also have them converted to grayscale for additional(minor) space savings.
image_max_size: 580, 725
grayscale_images: true
The default size (580,725--width,height) is for my nook STR. If people tell me what appropriate sizes are for other book readers, I'll include them in the comments.

calibre cover
The plugin has a new option on the Basic tab, "Update Cover when Updating Metadata". When checked, the calibre cover image will be updated everytime the other calibre metadata is updated (assuming EPUB). If there is a cover image in the EPUB, it will be used. If not, calibre will generate the usual 'from EPUB format' type cover, usually showing the title page.

Internal changes
The code for updating existing EPUBs to add new chapters has been completely rewritten. You shouldn't see any change from the outside. Other than supporting images, that is. :)

Priority of personal.ini sections has been more formalized. When downloading an EPUB from ficwad, the sections will take priority in this order, from least to most specific:

[defaults]
[www.ficwad.com]
[epub]
[www.ficwad.com:epub]
[overrides]

Which is why, if you set options in [epub], to override them for ficwad, you have to set them in [www.ficwad.com:epub]

For convenience of cutting and pasting, here's all the new image parameters together:

[epub]
## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover: true

## If set, the epub will never have a cover, even include_images is on
## and the site has specific cover images.
#never_make_cover: false

## If set, and there isn't already a cover image from the adapter or
## from make_firstimage_cover, this image will be made the cover.
## It can be either a 'file:' or 'http:' url.
## Note that if you enable make_firstimage_cover in [epub], but want
## to use default_cover_image for a specific site, use the site:format
## section, for example: [www.ficwad.com:epub]
#default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
#default_cover_image:http://www.somesite.com/someimage.gif

## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
image_max_size: 580, 725

## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: true

fimfiction.net locked story passwords
The downloader will now prompt for password on fimfiction.net's locked stories. Because each locked story has a different password, it's not useful to store a password in personal.ini. You can, however, use the fail_on_password option to prevent the plugin from prompting for story passwords.

[www.fimfiction.net]
fail_on_password: true


I appreciate hearing from both people with problems and people without, thanks.

v1.5.1 adds error handing for image loading, stay-on-top
v1.5.2 fixes a problem d/l'ing chapter text in GA.
v1.5.3 fixes a fimficnet problem, addes replace_hr & never_make_cover options.
v1.5.4 more fimfic fixs
v1.5.5 fimfic story password support & fail_on_password option
v1.5.6 add 'Include images in EPUBs' checkbox
v1.5.7 add 'nook_img_fix' - wrap img tags with div if not already wrapped with div or p for a nook problem. Defaults to true.

Jim

jackr
02-28-2012, 03:19 AM
I am very excited with these new updates :D. I am trying the beta out and so far everything seems to work except for when I use "Update Calibre Metadata Only". When I use that option, the cover gets overwritten with a screen-shot of the summary page.

And I noticed that it is set to inject the first image found into the epub, which I like the idea of for sending it to the Nook w/o having to do the reconvert method. But I think having an option as to whether the epub gets injected with the cover or not would be a good idea, if it is possible*

That's one thing I never understood about covers when I was working on an ebub in Sigil once. I was able to assign a cover image and keep it from being within the html pages. But it wasn't kosher, even though the Nook displayed the cover just fine.

JimmXinu
02-28-2012, 12:11 PM
I am very excited with these new updates :D. I am trying the beta out and so far everything seems to work except for when I use "Update Calibre Metadata Only". When I use that option, the cover gets overwritten with a screen-shot of the summary page.
Hmm. I think that's more a matter of labeling than an actual problem. The 'Update Cover' option tells calibre to take the cover from the epub. If the epub hasn't been updated and doesn't have a cover image, it does the usual default. Finding the 'first image' would require downloading the whole story again in the case of a story with no images--not exactly what we want for 'update calibre only'. :)
And I noticed that it is set to inject the first image found into the epub, which I like the idea of for sending it to the Nook w/o having to do the reconvert method. But I think having an option as to whether the epub gets injected with the cover or not would be a good idea, if it is possible*
If you have make_firstimage_cover off and don't have default_cover_image set, the epub won't have a cover in it. Is that sufficient, or do you mean something else?
That's one thing I never understood about covers when I was working on an epub in Sigil once. I was able to assign a cover image and keep it from being within the html pages. But it wasn't kosher, even though the Nook displayed the cover just fine.
:blink: Unfortunately, covers are not specified by the epub standards, so it's a ugly as different readers have different ideas. What I've implemented follows the 'best practices' I've been able to find.

Thanks for the feedback, it's very helpful!

jackr
02-28-2012, 10:46 PM
If you have make_firstimage_cover off and don't have default_cover_image set, the epub won't have a cover in it. Is that sufficient, or do you mean something else?

I tried removing make_firstimage_cover from personal.ini, but a cover is still injected into the epub.

What I mean about the cover is having it as an option so that only Calibre updates the cover. Before when I downloaded a new fanfiction and I either generated a new cover or pasted in an image, that cover wasn't injected into the epub. But maybe that is what you mean and I just can't get it to work yet :o.

Once again though, amazing work :). If I could send more karma pts, I would have. I will go hunting for some stories with images in them to test next.

JimmXinu
02-28-2012, 10:57 PM
I tried removing make_firstimage_cover from personal.ini, but a cover is still injected into the epub.
Is it a story from fimfiction.net? That site has specific cover images, so it gets set whenever the cover image exists and include_images is on. That is a case where a 'don't make a cover' option would be useful, I suppose.

If not, can you give me the story URL? And maybe a copy of your personal.ini--minus user/passwords, of course.
Once again though, amazing work :). If I could send more karma pts, I would have. I will go hunting for some stories with images in them to test next.
Thanks, glad you like it. :cool:

jackr
02-29-2012, 01:39 AM
Yes it was fimfiction stories that I tested with. My personal.ini is just this:
[defaults]
is_adult:true
#Accept-Encoding=gzip

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:true
I am not to clear on whether accept-encoding needs a colon or equals, so I have it on standby for now.

An idea that just popped in my head as a possibility for a custom column option is an error code or some sort for stories that have errors when updating. I've come across a couple that no longer exist, and just a bit ago one that had some message about news or some sort for it (its error icon was the red one with a white x, I will check later to see what the exact error was).

Edit:
The error code I got was:local variable 'newsrc' referenced before assignmentfor http://www.fimfiction.net/story/5947/.

Another error code I've gotten is:list index out of rangefor http://www.fimfiction.net/story/9948/.

As well as several:'NoneType' object has no attribute 'find' for fimfiction stories that have either disappeared from the site or have had password locks placed on them by their authors.

JimmXinu
02-29-2012, 12:19 PM
@jackr -

Re: cover image--the plugin gives fimfiction stories a cover whenever story has a cover image on the site and include_images is true. I'll add an option to suppress cover generation in general for such cases.

Re: Accept-Encoding--there's no such option in personal.ini, you can delete it.

Re: http://www.fimfiction.net/story/5947/ --downloaded fine for me with v.1.5.2.

Re: http://www.fimfiction.net/story/9948/ --Failed for me, too. I'll work on it.

Re: disappeared from the site or have password locks --can you give me an example URL for each?

Thanks for your feedback, it's very helpful.

JimmXinu
02-29-2012, 12:31 PM
Re: http://www.fimfiction.net/story/9948/ --Failed for me, too. I'll work on it.
Failed because there's no chapter in that story--it reports 0 words, too. I'll fix the downloader so it doesn't fail outright, but I'm not sure downloading an empty story is useful. :)

JimmXinu
02-29-2012, 01:52 PM
Beta version is now up to v1.5.3:

http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

v1.5.1 adds error handing for image loading, stay-on-top
v1.5.2 fixes a problem d/l'ing chapter text in GA.
v1.5.3 fixes a fimficnet problem, adds replace_hr & never_make_cover options.

Hopefully this is just about ready to go.

jackr
02-29-2012, 05:14 PM
I didn't notice the update to the beta, as well as the empty story (I just loaded it and noticed it had an image, didn't look at chapter list) X-).

Where is Accept-Encoding? I know you mentioned this feature a while back, but I couldn't find it elsewhere in the plugin. I probably misread your post about it :o.

http://www.fimfiction.net/story/9439/ is an example for a story that no longer exists.
And http://www.fimfiction.net/story/6884/ for a password locked story.
Both of these get the 'NoneType' object error.

I ran the plugin (v.1.5.3), and came up with a lot more errors than in 1.5. A majority of these new errors are:local variable 'datePublished' referenced before assignmentAlso the story that you mentioned worked in 1.5.2 has en error for me in 1.5.3. (http://www.fimfiction.net/story/5947/)

I've uploaded an image showing all the errors I got this time around. http://i39.tinypic.com/16k57o4.png

JimmXinu
02-29-2012, 06:28 PM
@jackr - Accept-Encoding=gzip is an HTTP protocol parameter used down inside the system when it's getting web pages from sites. It's an internal thing that you don't need to worry about.

Missing fimfic stories now handled correctly. It still errors, but at least it's a StoryDoesNotExist error now.

Wow--password per story? That's a new one. I did not know fimfic did that. I've tried to implement it using the user/pass mechanism. If it doesn't work, can you give me a story and it's password? That would help (PM would be fine).

Ah. Oops. The datePublished error is a mistake--the 'fix' for empty stories broke stories with an earlier start date. My bad. Wait, no it didn't--it revealed another, different bug. So, yay, I guess? :shrug:

As for http://www.fimfiction.net/story/5947/, not sure what to say--it still works for me. I suggest doing an 'Overwrite always' on that one and see if that works. Or an 'Add New', if you're paranoid.

v1.5.4 is up on the original beta post.
http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

Thanks for your help figuring these out.

jackr
02-29-2012, 07:32 PM
Yup overwriting the story fixed the issue, whatever it was :D. The password lock is rather new for me too. I will see if I can place a dummy story on the site and lock it, as I don't know any passwords to the ones that are locked.

Update:
I created a locked story on fimfiction, I will PM you with the details.

JimmXinu
02-29-2012, 07:52 PM
Interesting. If you see the newsrc problem again, I'll probably need a copy of the offending epub to track it down. Hopefully it was just an issue with the different beta versions that won't crop up again.

Locked story works for me, but I want to change the password dialog a bit so people aren't using their account user/pass.

I assume the other issues were all resolved.

Again, thanks for your help.

jackr
02-29-2012, 08:30 PM
Ah the error messages are much more legible now :D. I didn't encounter the newsrc error again.

As for individual story passwords, of the locked stories I have in my Calibre, I downloaded before the author locked them. By the looks of it, fimfiction is set up so an author can lock away their story if they feel it needs revision or whatever else. I think having a setting to disable password prompt for stories that don't have a defined password would be excellent.

JimmXinu
02-29-2012, 09:21 PM
Good suggestion.

v1.5.5 is up on the original beta post.
http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

jeaterod
03-01-2012, 01:28 AM
can you add http://www.adultfanfiction.net to the downloadable?

JimmXinu
03-01-2012, 02:51 PM
@jeaterod - Sorry, I'm not touching a site with that much legal warning on it.

http://anime2.adultfanfiction.net/form_adult.php

JimmXinu
03-01-2012, 03:02 PM
Okay, hopefully this really is the last beta version for this round.

v1.5.6 add 'Include images in EPUBs' checkbox

http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

If nobody's reported any problems with it by this evening(4-5 hours from now), I'll put it out for everyone.

Thanks to everyone who's been helping test this.

jackr
03-01-2012, 04:39 PM
This has only happened occasionally, and it might be something to do more with Calibre itself perhaps, but when I go to update fanfics and the process is on the 'Downloading metadata for stories' window, sometimes the process will stop and disappear when I go an load other windows.

I think it might be happening because I have set Calibre to be appear as an icon next to my clock. I noticed that the plugin will sometimes freeze and flash away and back between each story for metadata downloading. So maybe during that brief window, Calibre decides it can go in the background? :blink:

It's not a big thing, has only happened about 2 or 3 times now. And it seems to occur most during the beginning of the download process.

JimmXinu
03-01-2012, 05:12 PM
@jackr - I've never tried the system tray feature of calibre before.

I bet that you tried to minimize calibre to the system tray.

If I click the calibre system tray icon to hide calibre while it's downloading metadata, it aborts the plugin as if I'd clicked the close button on the progress bar.

I see the same behavior from Generate Cover, so I expect any in-progress action that can be cancelled, will be. Copying from one library to another doesn't get cancelled, but neither does it cancel when you click it's close button.

So I'm going to call it a general calibre behavior. I think being able to abort the download is more important than hiding to system tray.

jackr
03-01-2012, 07:07 PM
I had a feeling it was more of a Calibre thing. And I probably did try to minimize without thinking. :) So far the plugin is working solid.

Is it possible to combine the cover with the jacket inside the epub? Like having it between the title/author and other metadata/summary? Or would that not work with the jacket containing just metadata info?

JimmXinu
03-01-2012, 07:13 PM
EPUB best practices say that the cover image has to be just an image. There's already a very nice Generate Cover (http://www.mobileread.com/forums/showthread.php?t=124219) plugin that will create an image with title, etc on it. And you can choose '{Current Cover}' to combine with the existing cover image.

Does that help you any?

jeaterod
03-01-2012, 07:22 PM
aww thats bs legal warning tho... guess ill just have to keep using flagfic to get downloads from that site

jackr
03-01-2012, 07:27 PM
I will find a way to make do, thanks for the reply :). I hope that epub evolves a little in the future, I just find this idea so appealing for some reason :o: http://i42.tinypic.com/qxm3uw.png

JimmXinu
03-01-2012, 07:59 PM
Hmm. I guess I didn't understand what you were asking for.

Inserting the cover image into the title page isn't unreasonable. Not in tonight's release, but I'll look into it.

JimmXinu
03-01-2012, 08:29 PM
FYI, v1.5.7 has been released.

jeaterod
03-02-2012, 05:07 AM
is there a way for me to tweek the plugin so that when it get the chapter number the numbers 1-9 become 01-09?

JimmXinu
03-02-2012, 01:10 PM
The plugin doesn't number the chapters. Any chapter numbers are coming from the source site. So, no, not really.

cklls
03-02-2012, 02:48 PM
Just wanted to say thanks again, Jim. This tool keeps getting better and better, and makes my obsession easier to support. Um... yeah. ;) :blush:

JimmXinu
03-02-2012, 03:51 PM
Glad I could be an enabler for you, cklls!

Wait... :uhoh:

:D

jeaterod
03-02-2012, 06:37 PM
so you cant tell the plugin to look at it and if the value is lower than 10 add 0 to the front?
....

JimmXinu
03-02-2012, 06:53 PM
"Prologue"
"Chapter 1"
"Mission Briefing"
"Chapter One"
"Chapter I"
"Ch 2"
"3rd Chapter"
"7. Girl Talk"
"9. Chapter 9"
"‘Problems in Engineering'"

These are all chapter names from various stories on sites. It's a free form text value, not just a number. And it's not uncommon for file0001 to be 'Prologue' and file0002 'Chapter 1'. Or '2. Chapter 1', depending on the site.

I could give you an option like replace_metadata in personal.ini to modify chapter titles with a regular expression replacement, but you'd have to work out the expression to your own satisfaction.

jeaterod
03-02-2012, 07:44 PM
oh sorry i dont mean in the story it self i mean in the library colomns

JimmXinu
03-02-2012, 07:58 PM
Oh, you don't mean chapter numbers, you mean the number of chapters.

You want to use an integer custom column and then set the format to: {0:02d}

jeaterod
03-02-2012, 08:03 PM
ok ill try that

EDIT:
awesome thanks it worked great

animealam
03-04-2012, 09:33 PM
First of all, thanks for the plugin, it's awesome.:2thumbsup:thumbsup:

I read some of the info you have for adding a site, but frankly python is way over my head. I was going to ask for Adultfanfiction.net, but saw you won't add it sadly. It is completely adult oriented, hence the over abundance of warnings. :(

Anyway, could you maybe add HP Fandom.net (http://www.hpfandom.net/eff/index.php)?
It's an old eFiction based site (1.1 I think) and it's a large archive, actually one of the largest all Harry Potter out there.

Then there is Walking the Plank (http://www.walkingtheplank.org/archive/).

Hopefully you'd be able to add them.

JimmXinu
03-04-2012, 09:43 PM
First of all, thanks for the plugin, it's awesome.:2thumbsup:thumbsup:
You're welcome. :)
Anyway, could you maybe add HP Fandom.net?
It's an old eFiction based site (1.1 I think) and it's a large archive, actually one of the largest all Harry Potter out there.

Then there is Walking the Plank, and The Silver Snitch II, both are eFiction sites.
First, I suggest including the URL when requesting sites.

Second, unless it's something that's personally interesting to me, I'm working on other things than new sites lately. But we do also have other developers who might be interested.

iatheia
03-04-2012, 11:57 PM
First of all, thanks for the plugin, it's awesome.:2thumbsup:thumbsup:

I read some of the info you have for adding a site, but frankly python is way over my head. I was going to ask for Adultfanfiction.net, but saw you won't add it sadly. It is completely adult oriented, hence the over abundance of warnings. :(

Anyway, could you maybe add HP Fandom.net (http://www.hpfandom.net/eff/index.php)?
It's an old eFiction based site (1.1 I think) and it's a large archive, actually one of the largest all Harry Potter out there.

Then there is Walking the Plank (http://www.walkingtheplank.org/archive/).

Hopefully you'd be able to add them.

Eh, I may look at the first one probably next week. Not quite right now. (though, the warning on the front page is not particularly inspiring)
The second one is a bit on the small side, though.

Zauberlehrling!
03-05-2012, 04:52 AM
Hi, just tried the plugin for the first time and have a question already.
Is there any chance to get http://ksarchive.com (Fanfiction Site for Kirk/Spock Stories) included into that plugin, please?

jackr
03-05-2012, 07:00 PM
An idea crossed my mind for password locked stories. If the story is locked, the plugin goes to check a password section in personal.ini to see if it has a password there, and if so it tries to use the password. That way if there are a few locked stories that a reader is privy to, they can still access those stories without needing to wait for the password prompt for all the other locked stories that they aren't privy to.

Something like:[Password]
fimfiction.net/story/796/:*password
fimfiction.net/story/6884/Moonlight-Sonata:12345

And if there is a password listed, but it doesn't match, then maybe have so the error code reflects that.

Just a thought that cropped up when I was working on a story that I have locked. I was going to try out an image inside the story, but was too lazy to go remove the fail_on_password for it. :D

Jade Aislin
03-05-2012, 09:44 PM
I had started to work on an adapter for ksarchive.com, however I hit a snag when I realized that it has two different warnings, which none of the other sites have. I have almost no experience with python and was not sure how to continue. Unfortunately, because of life and school, I have had little time to work on it either.

JimmXinu
03-05-2012, 09:48 PM
Actually, adapter_tenhawkpresentscom.py has two different warnings--depending on how you mean 'warning'. Have you looked in that one?

Jade Aislin
03-05-2012, 09:51 PM
No, but I will look at it tomorrow. School calls tonight. If I remember, ksarchive had a warning = 1 for teen rated stories and a warning = 2 for adult rated stories. I'll look at the adapter_tenhawkpresentscom.py. Hopefully it will solve my problem.

JimmXinu
03-05-2012, 09:53 PM
Something like that. It requires warning=3 for some stories and warning=4 and log-in for others.

JimmXinu
03-06-2012, 12:01 AM
I'm looking for some testers to help find the first round of bugs with four new sites.

fanfiction.mugglenet.com
fanfiction.portkey.org
thequidditchpitch.org
www.hpfandom.net

These are the four largest/most active of the sites that have been requested lately.

Another site that's been requested a few times is adultfanfiction.net. None of the current developers are interested in writing code to support it because of the ominous legal warnings and requirements to access the site.

JimmXinu
03-06-2012, 10:23 PM
Nobody's complained--must work fine. Advanced to general release.

SheDo2
03-07-2012, 11:49 PM
I just started using FanFictiondownloader plugin and it's great... Could you please add a couple of my favorite sites...
http://www.academyofbards.org/

http://www.xenafiction.net/

JimmXinu
03-08-2012, 12:05 AM
@SheDo2 - Probably not.

Those two sites, at first glance, are really just websites that happen to contain stories.

What makes it possible to reliably convert stories to ebooks is the fact that the supported sites use story management software to present the stories in a consistent way.

I suggest googling for 'web page to epub' or something along those lines. There are a lot of different solutions for websites, I believe.

iatheia
03-08-2012, 12:06 AM
I just started using FanFictiondownloader plugin and it's great... Could you please add a couple of my favorite sites...
http://www.academyofbards.org/

http://www.xenafiction.net/


Took a quick look... both of these websites have very, very ugly html formatting, with pretty much every story having a free-form layout. There isn't really a place to easily collect even something as simple as a table of contents for a multi-chaptered work, not to mention metadata. Hell, you can't even get author url from any of the stories!
While not _impossible_, I wouldn't really get your hopes up.

SheDo2
03-08-2012, 07:53 AM
Thanks for the reply. I use iSiloX as a drag and drop HTML converter but it only converts to iSilo format and Calibre doesn't convert iSilo. Oh well back to going around my elbow to get to my a$$...

Thanks for lookin at my request.

Gin

davidfor
03-08-2012, 09:32 PM
Thanks for the reply. I use iSiloX as a drag and drop HTML converter but it only converts to iSilo format and Calibre doesn't convert iSilo. Oh well back to going around my elbow to get to my a$$...

If you are using Firefox, have a look at the extension GrabMyBooks (http://www.grabmybooks.com/). It allows you to build epubs from web pages. I tried it a while ago out of curiosity, but I don't remember how good it was.

Jade Aislin
03-09-2012, 02:32 AM
Something like that. It requires warning=3 for some stories and warning=4 and log-in for others.

I looked at tenhawkspresentscom.py. It looks like it has one regular warning and one warning for stories that need a registered user. I looked at the other sites that have adapters. Now I'm completely confused. From what I could tell, all the sites had at least one warning that needed the user to be registered when there was more than one warning.

ksarchive does not need the user to be registered. You only need to go through the javascript pop up for two different ratings.

I do not understand python at all and can't seem to get beyond editing the basic things like it is outlined in the castlefanorg.py. That is how I was able to get the lower rated stories to work before. Unfortunately, I have too many other things going on in my life at the moment and don't think I will be able to continue working on this.

I had tried to use
if "text from javascript" in data:
addurl= (the warning # depeneding on the text)

But it still didn't seem to be adding the warning to the url. I also was not sure where to put the lines as I ran into errors in both the data and url not being defined.



I do want an adapter for ksarchive.com. But I have no idea what I am doing. Unless someone else does one similar, I do not think I can continue.

- Jade Aislin :blink:

JimmXinu
03-09-2012, 11:23 AM
@Jade Aislin - That's fine. It's on our list of sites to consider and will likely get supported sooner or later.

summerdream81
03-09-2012, 02:50 PM
Hello,

I'm new to using the plugin and I really love it. Thanks for all the great work !!

And now I have, maybe a little dumb questions :

How exactly do I configure personal.ini ?
I tried to add it in .config>calibre>plugin>fanfictiondownloader.zip but somehow I still have to put in my logins manually?
How do I have to format it exactly ? do the ## and # from your example stay ?

And when I try to download from twiwrite I only get errors?
'NoneType' object has no attribute 'string'

I tried this story http://www.twiwrite.net/viewstory.php?sid=785&ageconsent=ok&warning=1
and some others and its always the same error


Maybe someone can help?

TY summerdream81

JimmXinu
03-09-2012, 03:03 PM
@summerdream81 - I think you're trying too hard. :)

Drop down the FFDL plugin menu, choose 'Configure Plugin', choose the personal.ini tab. Edit away. Or, do 'Preferences' > 'Plugins' > 'User Interfact Action plugins' > FanFictionDownLoader

.ini config file format is pretty basic. [sections] and keyword:value lines. The authoritative format source is: http://docs.python.org/library/configparser.html

# and ## are both comment lines and maybe removed if you like. As a convention, I try to use # in front of keywords that FFDL knows that can be turned on by removing the # and ## for comments about them.

As for http://www.twiwrite.net/viewstory.php?sid=785, it downloaded fine for me.

summerdream81
03-09-2012, 03:35 PM
@JimmXinu
Wow thank you for the fast repliy.
Ahh so simpel !! thats great, now the login is working smoothly. TY very much.

But I still get the same error every time I try to download from twiwrite?

JimmXinu
03-09-2012, 03:49 PM
@summerdream81 - I can probably help if you get me some more information.

Run 'calibre-debug -g' from a command line, try downloading from twiwrite again and copy-paste the command line output here. It should contain more error info.

summerdream81
03-09-2012, 04:16 PM
Okay here is the debug protokol :

calibre Fehlerdiagnoseprotokoll
calibre 0.8.41
Linux-2.6.38-8-generic-x86_64-with-debian-squeeze-sid
Linux
('Linux', '2.6.38-8-generic', '#42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011')
Python 2.7.1
Linux: ('debian', 'squeeze/sid', '')
Starting up...
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Started up in 16.53 seconds with 2173 books
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
FanFictionDownLoader v1.5.8
url:http://www.twiwrite.net/viewstory.php?sid=785
Exception: {u'comment': u"'NoneType' object has no attribute 'string'", u'good': False, u'author': u'Unknown', u'url': u'http://www.twiwrite.net/viewstory.php?sid=785', u'author_sort': u'Unknown', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'string'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 262, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 428, in get_metadata_for_book
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 225, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.adapter_twiwritenet", line 131, in extractChapterUrlsAndMetadata
AttributeError: 'NoneType' object has no attribute 'string'
Fetched metadata for 1 of 1

I hope this helps

JimmXinu
03-09-2012, 05:47 PM
I think I've figured it out.

Try going into your account on twiwrite.net, Edit Preferences and change "I am of the age of consent in my locale..." to Yes.

I'll add an 'is adult' check to that adapter for that case, but it wouldn't be right away.

crochetgeek2010
03-09-2012, 09:13 PM
Any chance of adding http://www.squidge.org? thanks Laura

JimmXinu
03-11-2012, 01:13 AM
@crochetgeek2010 - Unlikely. It's not just a story site, it's several collections of differently formatted, differently organized stories and pages.

It's one of those sites that's bad enough that I doubt anyone who doesn't read it themselves is going to be interested in writing the adapter.

Zauberlehrling!
03-11-2012, 10:41 AM
I had started to work on an adapter for ksarchive.com, however I hit a snag when I realized that it has two different warnings, which none of the other sites have. I have almost no experience with python and was not sure how to continue. Unfortunately, because of life and school, I have had little time to work on it either.

Oh super, thanks for trying to "crack" this archive. I know what you mean with two different warnings. I don't know why they have it. Is it possible for you to keep trying?

cordy
03-11-2012, 11:21 AM
I had started to work on an adapter for ksarchive.com, however I hit a snag when I realized that it has two different warnings, which none of the other sites have. I have almost no experience with python and was not sure how to continue. Unfortunately, because of life and school, I have had little time to work on it either.

Hello, I'm no master, and I don't know if the coding I put together will be approved or not, but I tried to work around a site with 3 warnings, locally I'm able to get the fictions downloaded. If you want I can send you the bit of code I used ... maybe together we will be able to work around the two warnings on your website.

Sam.

jackr
03-15-2012, 02:23 AM
I forget if I asked about it before, but could you include in the Custom Columns section an option for outputting the various errors for fanfic's? I feel like I did bring it up in an earlier post, but a brief search made me think I probably forgot to ask. :o

I know one benefit I would get from it would be color coding stories that do get errors as well as filtering them out if I need to run an overwrite always on a couple stories, as well as just keeping general tabs on the various changes; like one fic recently that got a chapter count error because the author revamped their story and compacted chapters together, etc.

JimmXinu
03-15-2012, 10:11 PM
@jackr--That sounds familar, so I expect you did ask before.

This isn't something I'm likely to implement. The internal structure would have to be revamped considerable, plus there's the issue of what to do with errors on new stories that don't have existing books.

IMO, you shouldn't be using update to check for new chapters. That's what the update emails from the sites are for.

JimmXinu
03-16-2012, 06:37 PM
FYI.

Version 1.5.9 - 16 Mar 2012
Remove site gayauthors.org following complaints from the site administration.
New Site: nfacommunity.com
New Site: www.midnightwhispers.ca. Thanks to Sam for implementing these.
Make 'Update Calibre Only' add a new book if no matching book found.
Minor improvement to column updates for new stories.
Improve image support for a couple of obscure cases.
portkeyorg.org only - Write 'Chapter does not exist' chapter when chapter is in list, but does not exist.
Add is_adult check to twiwrite for when user on site is not set to adult.
Change td in portkey chapter text to div for nook.

cordy
03-19-2012, 09:07 AM
Hello,

I have a little problem, I'm running the 0.8.43 calibre portable edition with the plugin activated.

It worked great, until this WE. I didn't change anything, I upgraded to the new version. At first it didn't work, but then I thought maybe it's because the non portable version on the home computer wasn't up to date. I did the update.

And it didn't change a thing, I still had the problem.

When I use the plugin, it grabs the metadata, but doesn't dl the fic.

I tweaked both installs at home, and I was finally able to get it to work. But now, I'm only on the portable version, and the plugin do the same.... grabs the metadata, but not the story itself.

When I look into the "show job details" I have this :

Starting job: Downloading FanFiction Stories
Downloading None
Failed Downloading None: 'title'

doesn't seem too good.... does it?

The fiction I'm trying to grab is here : http://www.fanfiction.net/s/5846760/1/My_Fathers_Keeper


Oh and another problem ... I can't configure the plugin this is the error I have....
Can't understand why

calibre, version 0.8.40
ERROR: Unhandled exception: <b>TypeError</b>:__init__() takes exactly 1 argument (2 given)

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 138, in config
File "site-packages\calibre\customize\__init__.py", line 150, in do_user_config
File "calibre_plugins.fanfictiondownloader_plugin.__init __", line 68, in config_widget
TypeError: __init__() takes exactly 1 argument (2 given)


Thanks for any idea.
Sam

JimmXinu
03-19-2012, 12:17 PM
@cordy - I don't think you truly have the latest version of either calibre or the plugin.

The error you show at the end says calibre 0.8.40, not 0.8.43. But that shouldn't make any difference for this problem.

The plugin writes it's version to the calibre job log and has for a number of versions now. It should look like this:
Download FanFiction Book
FanFictionDownLoader v1.5.9
Adding jobs for URLs:

Plus, the line number in the plugin your exception post references is blank, again indicating version mismatch.

If your plugin version is that old, you're probably bumping into the format change ffnet made a while back.

I'm not familiar with portable installs, so I can't help you with that part.

Brown Betty
03-19-2012, 02:31 PM
Hey, thanks for all of your work on this amazing plugin.

I was pleased to see tth added to the list of supported sites, but trying to download http://www.tthfanfic.org/wholestory.php?no=18987#chapter1 fails silently, for me.

Running calibre-debug -g from the terminal produces almost no output (can this be right?) but here is what it does give:

Starting up...
Started up in 3.37 seconds with 822 books
FanFictionDownLoader v1.5.9


And nothing appears in the job log.

JimmXinu
03-19-2012, 03:58 PM
The plugin silently discards URLs (and other text) that it doesn't recognize as story URLs.

To properly chapter a story, you have to use the regular story URL, not the wholestory URL. http://www.tthfanfic.org/Story-18987/jedibuttercup+Handle+With+Care.htm works fine.

JimmXinu
03-19-2012, 10:41 PM
FYI, New version out with these changes:

- New site: ksarchive.com, thanks to Jade Aislin for implementing this.

- Fixes for mugglenet.com--added user/pass, better handling for summary and is_adult levels.

- Allow default_cover_image to use ${title}, etc; add cover_exclusion_regexp feature to exclude images from being covers.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

cordy
03-20-2012, 03:41 AM
@cordy - I don't think you truly have the latest version of either calibre or the plugin.

The error you show at the end says calibre 0.8.40, not 0.8.43. But that shouldn't make any difference for this problem.

The plugin writes it's version to the calibre job log and has for a number of versions now. It should look like this:
Download FanFiction Book
FanFictionDownLoader v1.5.9
Adding jobs for URLs:

Plus, the line number in the plugin your exception post references is blank, again indicating version mismatch.

If your plugin version is that old, you're probably bumping into the format change ffnet made a while back.

I'm not familiar with portable installs, so I can't help you with that part.

Thanks for helping me... I battled against the plugin for 10 minutes and you where right, it was the very first version installed with the latest overwritten. In fact it was installed twice, and the worked against each other.

Now it's fixed, and your plugin works great on the portable calibre edition.

Thanks
Sam

jackr
03-25-2012, 12:34 AM
Haven't gotten a response back from the site owner of Fimfiction.net after sending them a message a couple weeks ago; I will try sending another soon.

I know that the posts (if there are any) do get dated with a year, but that of course is sketchy for basing the year on. And the stories from the author's creation dashboard (or whatever is a good word for that) have a year on them for "Last Modified", but that doesn't help any if it is not there in the target html. :(

Oh well, it's ultimately a small thing. Anyways, keep up the awesome work! :thanks:

JimmXinu
03-25-2012, 11:24 AM
@jackr - I expect they'll start thinking about it when the site gets old enough that the story dates become ambiguous.

cryzed
03-25-2012, 05:51 PM
Hey there JimmXinu, I just found out about your plugin for Calibre and it is really looking good so far :)! The thing is, I wrote my own solution for FanFiction.Net and FictionPress.com quite a while ago, it's called Lemon (https://bitbucket.org/cryzed/lemon/overview).

After inspecting your adapter files for the respective sites I thought maybe you could use the two APIs I wrote for them: FanFiction.Net API (https://bitbucket.org/cryzed/fanfiction.net-api/overview) and FictionPress.com API (https://bitbucket.org/cryzed/fictionpress.com-api/overview). They don't introduce any new dependencies besides html5lib (using html5lib in correlation with BeautifulSoup is a good idea anyhow) and even easily parse information that you are currently missing, e.g. the review count (which I would really like to see in the final file if possible).

And I realize, of course, that this is only personal preference but could you adapt the default template to something like this (http://www.imagebanana.com/view/fs0ptlz5/screen_shot65231.gif)/this (http://www.imagebanana.com/view/w3g4o9uv/screen_shot65174.gif)/this (http://www.imagebanana.com/view/oumsjyqz/screen_shot65193.gif)? The source for the templates used by Lemon can be found here (https://bitbucket.org/cryzed/lemon/src/8840a93d0bd8/lemon/fanfiction_net/template.py) and here (https://bitbucket.org/cryzed/lemon/src/8840a93d0bd8/lemon/fictionpress_com/template.py). Only if you agree, of course, but quite honestly (no offense): I really think that the templates used by Lemon are an improvement over the ones currently used in the plugin.

Man, I sound like a smug asshole -- but this really isn't my intention, I just want to finally deprecate Lemon for something better and I would be all over your Calibre plugin if you made these small changes.

JimmXinu
03-25-2012, 11:14 PM
@cryzed -

I'm not really inclined to change FFDL to use your APIs for ffnet and fpcom. If you were offering code for a new site, or significant new functionality, it would be a different matter.

As for review count, I don't see that the value is worth the effort. It's going to be out of date most of the time. FFDL has 25+ different adapters for different sites it would have to be added to. And if FFDL collects review count, why not the recs/favorites/hits counts that many sites have, too?

I've added it to the 'wishlist' of future features, but I don't see it happening any time soon.

(BTW, to counter the argument that review counts could be updated everyday: I strongly recommend everyone use email notices for new chapters, and to not poll all your stories every day for new chapters. It inflates hit counts and wastes bandwidth on the fanfic site. We can only do this as long as we don't annoy the sites enough to block us.)

As for changing the appearance of the output, looking at your screen shots, the only differences between Lemon and FFDL output I noticed are that Lemon doesn't put commas in the word count and it centers the TOC entries. And, predictably, I like the output from FFDL better--because it's the way I like it. :)

Have you looked at FFDL's personal.ini settings? Specifically, the titlepage_entries, include_tocpage, and output_css options? There's already a fair degree of customization available. (Fair warning--FFDL mobi output doesn't support output_css. In fact, FFDL mobi output is only one step above deprecated. I strongly recommend epub and conversion.)

If you have specific requests, I might be able to add some additional markup to the title and TOC page output for the CSS to influence.

Thanks for your interest.

nomad062
03-26-2012, 03:12 AM
I'm playing around using Ubuntu and of course one of the first things I downloaded was Calibre - the second thing I wanted was your FFDL plugin. Awesome plugin. Saves me several steps now that I can do it in 1 shot.

At any rate when I went to download your plugin it states that I need Calibre 8.30 or later yet my Calibre says its 8.8. Any idea? or do I need to post something with Calibre?

PeterT
03-26-2012, 09:06 AM
Do an install of calibre from the official web site (http://calibre-ebook.com/download_linux), and not from the Ubuntu repository.

nomad062
03-26-2012, 11:38 AM
Thanks for the info!

JimmXinu
03-26-2012, 12:24 PM
What he said.

0.8.30 is specified because that's the earliest version I've tested it with. There's probably at least a few earlier versions it would work with, but 0.8.8 is too far back--there were significant changes to plugin support around 0.8.18.

nomad062
03-26-2012, 01:07 PM
I guess I was misreading the 8.8 I thought it was more recent than 8.3 but its all fixed now so thank you both for the quick response

Laurin

cryzed
03-26-2012, 02:02 PM
@JimmXinu all right, I did it myself then -- while I was it I also implemented support for special title entries that allow a degree of customization (e.g. br-tags).

I still think that your code in its current form is really bad: in nearly all your files you have unused partial or even full module imports (sometimes duplicated), nothing that resembles PEP-8, you do some really puzzling things with the encoding/decoding of UTF-8 strings in your code and rely on chardet to guess encodings at times (although I didn't discover where), you use BeautifulSoup but apparently have no clue how to navigate its structure, RegEx is a solution for everything in your code while only making things more complicated (e.g. using it to strip '#' at the end of a string instead of just splitting the string), many of your methods do unexpected things, you use decorators incorrectly (e.g. adapaters.adapter_fanfictionnnet.FanFictionNetSite Adapter.getAcceptDomains), you have many methods that have a "get"-prefix and simply return a constant or unchanged value... and that's only after looking at 3-4 files of your project. That's why I wanted you to use my API modules, many, if not all things, you currently do in the adapter for FanFiction.Net and FictionPress.com could have been simplified.

Things that have me stumped are also that somehow when adding new items to the validEntries list and a corresponding label to the titleLabels list in your base_writer.py code the label is not used like apparently intended in BaseStoryWriter.writeTitlePage and the "NO_TITLE_ENTRY" is used as a template, so one has to actually manually define a "<title_entry>_label" entry in the configuration files -- why do the "fall back labels" exist then in the first place? (I'm sure this one's my fault) Additionally modifying the EPUB_TOC* constants in the EpubWriter class somehow has no effect on the resulting table of contents that is actually created, why is that?

Of course, I realize that my code is surely not perfect either but reading your code is really hard and puzzling -- and that's not because the things you do are complicated: the overall structure is just bad. This time there's no way that I could not have sounded like an asshole, sorry.

JimmXinu
03-27-2012, 10:31 PM
@cryzed - I'm not going to argue with you about it. I'm not a python expert, nor have I claimed to be. This is a hobby.

If you'd like to contribute any of your changes, please set up a clone (http://code.google.com/p/fanficdownloader/source/clones) to check your changes into and the other developers on the project and I will review them.

jackr
04-09-2012, 05:00 AM
I was busy tweaking the look of the Book Details pane today when I learned about how links can be assigned to author names. If you are looking for any new ideas at the moment, having the author link direct to the authors page would be great. :D

davidfor
04-09-2012, 07:41 AM
I was busy tweaking the look of the Book Details pane today when I learned about how links can be assigned to author names. If you are looking for any new ideas at the moment, having the author link direct to the authors page would be great. :D

In the tag browser, right click on the "Authors" category and select "Manage Authors". The third column in the list is what you want. I don't know how long that has been there, but, I only discovered this in the last couple of weeks.

JimmXinu
04-09-2012, 10:45 AM
I didn't know that was there either--I'll certainly try and populate it with the author url.

Thanks to both of you for pointing that out.

FFreader920
04-09-2012, 03:15 PM
I've been using the FFDL web app for some time now and was very interested in this plug-in. However, I can't get it to work for me at all. Opening the app, clicking a menu, touching anything immediately shuts the plug-in down (along with Calibre). I posted in another forum earlier and was directed here. Here's part of the error. I've got more if you want. Any thoughts? I updated Calibre version and just DL'd plug-in update.

Thanks for your help.

---
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000008

VM Regions Near 0x8:
-->
__TEXT 0000000100000000-0000000100002000 [ 8K] r-x/rwx SM=COW /Applications/calibre.app/Contents/MacOS/calibre

Application Specific Information:
objc[1124]: garbage collection is OFF
Performing @selector(qtDispatcherToQAction:) from sender NSMenuItem 0x110e395a0

---

JimmXinu
04-09-2012, 03:24 PM
@FFreader920 - Actually, I was trying to direct you up a level or two where the calibre devs (vs plugin devs) hang out. I don't know any more here than I did on the mail list, sorry.

FFreader920
04-09-2012, 04:16 PM
@FFreader920 - Actually, I was trying to direct you up a level or two where the calibre devs (vs plugin devs) hang out. I don't know any more here than I did on the mail list, sorry.
No problem -- and thank you! I will head there.

JimmXinu
04-10-2012, 11:02 AM
@FFreader920 - Following Kovid's suggestion, I've made a version that keeps explicit references to the menu items. Please give the attached version a try and see if it solves your problem.

See the Plugin Intro thread (http://www.mobileread.com/forums/showthread.php?t=118680) if you need help manually installing it.

FFreader920
04-11-2012, 01:33 AM
@FFreader920 - Following Kovid's suggestion, I've made a version that keeps explicit references to the menu items. Please give the attached version a try and see if it solves your problem.

See the Plugin Intro thread (http://www.mobileread.com/forums/showthread.php?t=118680) if you need help manually installing it.

Wow -- so ... that absolutely worked! I just downloaded a couple of stories and updates. Thank you so much. :thanks: It's like magic!

JimmXinu
04-11-2012, 10:50 AM
I'm glad we could get that sorted. I hope to put it out as the next official version later today.

JimmXinu
04-11-2012, 12:17 PM
FYI, v1.5.13 is out.

New site: archive.skyehawke.com plus a fix for FFreader920's crashing problem.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

jackr
04-14-2012, 10:29 PM
It looks like fimfiction.net has undergone some changes again. When updating, I am losing the cover images on most of the stories. So far only one has kept itself after an update.

http://www.fimfiction.net/story/6975/ This one still works.

http://www.fimfiction.net/story/4843/ And this is an example of one that doesn't.

I noticed on one fic that the author included a source link beneath their image. I never seen this before, so maybe that has something to do with it? :blink:

http://www.fimfiction.net/story/19066/ Has a source link.

And last but not least, is there a better place that I should go for adapter-site issues? I don't want to clutter this thread with posts about issues if they would be better addressed elsewhere. :o

JimmXinu
04-14-2012, 11:10 PM
@jackr - The project user group (http://groups.google.com/group/fanfic-downloader) is arguably a better place for adapter issues, as opposed to plugin specific issues. However, it's not always clear when an issue is plugin specific or not. If you found the problem using the plugin, I'm not likely to complain if you post it here.

This particular problem with fimfiction.net is happening because fimfiction.net is leaving off the 'http:' on story image URLs. IE, '//static.fimfiction.net/images/...' instead of 'http://static.fimfiction.net/images/...'. The browser are apparently prepared to live with this level of broken and fill in the 'http:'.

(http://www.fimfiction.net/story/6975/ only appears to work because there is a similar image in the first chapter that's being picked up for the cover.)

I've put a kludge in the adapter to deal with this and attached a beta version for you to try. I won't be able to put out an official version until Monday.

You could try reporting it as a problem to the site admins, but since they don't seem to think dates need the year...

jackr
04-15-2012, 01:01 AM
Thanks, works perfectly so far :D. According to a post on their main page, fimfiction has undergone a server change and that there are some bugs to work out. I will post a comment on their site about that partial url issue.

Amalthia
04-16-2012, 02:26 AM
Hi, I love this plugin and I just had one question. Is there a way to disable the confirmation box that pops up asking if you're an adult for the archive of our own archive? It's not so bad when I'm downloading one story but when I try to download five or six at a time it's not so much fun.

I'd appreciate any and all help,
thanks!
Amalthia

EDIT: Figured it out. Apparently I needed to remove the # sign from the url, username, password, and the adult:true line.

JimmXinu
04-17-2012, 03:14 PM
New version released:

Change "Include images in EPUBs?" to effect [epubs], not [defaults].
Kludge for incomplete story image URLs on fimfiction.net

jackr
04-19-2012, 04:52 AM
Fimfiction.net is down for maintenance by the looks of it. But I have found out that if I run an update on any stories that contain them, the plugin hangs and I have to force quit Calibre. I ran a test on a single fimfiction story and the time it takes to determine whether it can get the story is long; I counted 84 seconds that the plugin waits for to get a response.

The error code I get after the wait is this:<urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

I am not sure if it will help any with the plugin, and thankfully sites are rarely down, but I am posting this in case it is something that can be tweaked so that the plugin doesn't hang on a story more than it has to.

JimmXinu
04-19-2012, 12:16 PM
@jackr - Because the server doesn't return a connection refused, the downloader has to first wait for the connection to time out. Then, to deal with slow & flakey servers, it sleeps a little while and tries again a couple times before really failing.

It's a trade off between failing quickly when it's really down and waiting & trying when it's just flakey. I'm inclined to leave it as it is.

Ajax
04-21-2012, 07:19 PM
Finally I can download FictionPress books and not be forced to read online. Thank you so much!

JimmXinu
04-21-2012, 07:21 PM
@Ajax - Glad you like it.

Just M.E.
04-21-2012, 10:14 PM
I really love your plugin; it's made my fanfiction collecting so much easier! However I've run into a slight problem.

In the last few days, the Archive of Our Own website changed their layout to include different banners. Even though there is an option to hide the banner, whenever I use the plugin to grab a story, the title always comes up as 'Hide this banner', whether I am updating or getting a new story.

Is this something that can be changed in the plugin, or can you suggest a workaround other than editing every title manually?

Thanks in advance.

M.E.

JimmXinu
04-21-2012, 10:23 PM
@Just M.E. - FFDL will need to be updated.

Can you test the attached version and make sure it works for your stories? I don't read Archive of Our Own much.

Just M.E.
04-21-2012, 10:36 PM
@Just M.E. - FFDL will need to be updated.

Can you test the attached version and make sure it works for your stories? I don't read Archive of Our Own much.
Perfect, Thank you so much!

M.E.

JimmXinu
04-22-2012, 11:20 AM
v1.5.15 posted with changes:

Fix for 'Hide this banner' title issue on Archive of Our Own.
Changed implementation of fimfiction to use provided API, thanks to Althaine.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

pookakitten
04-24-2012, 04:05 AM
I have a story from a site that disagrees with my Kindle Touch. I have it set to download to mobi, but when I read to a location, it won't let me read further. Some of the text is superimposed on itself when I reach the part that won't let me advance. Any text after this point is not visible. This happens near location 525 (in chapter 6) on the Touch. I am using ver. 0.8.44 of calibre and the most current version of the plugin. The story works properly on the Touch if I reconvert it in calibre. I may have had a similar problem with another file from AO3.

Link to story: http://www.archiveofourown.org/works/299460

Thanks for the great software.

iatheia
04-24-2012, 12:33 PM
That's exactly it - download it in epub - it is much superior to mobi in every single way. Mobi is... wonky and is not supported properly.

JimmXinu
04-24-2012, 02:10 PM
Regrettably, that's true. FFDL's mobi output is far less than stellar. I spent most of a week once trying to improve it and ended up with something worse than before.

I download to epub and convert to mobi for my own kindle and that's what I recommend for everyone.

pookakitten
04-24-2012, 03:58 PM
Thanks for the heads up. I'll do that in the future.

pookakitten
04-25-2012, 07:52 PM
Hey, some more AO3 wackiness. I'm using the latest version of both calibre and the plugin. The other AO3 stories I have work just fine. Just these 2 black sheep.

I've enclosed a screenshot with the error message and links that are acting up.

JimmXinu
04-25-2012, 11:05 PM
v1.5.16 posted with change:

Fix for Yet Another numeric entity issue for pookakitten's problem.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

JimmXinu
05-02-2012, 09:53 PM
Anybody interested testing a new feature? Namely, automatically calling Generate Cover(GC) on FFDL download/updates. I know somebody asked for it once. :)

There's a new config tab in FFDL for Generate Cover and three ways to set it up:

Easiest: Select GC setting for 'Default' to use for all downloads/updates.
Still Easy: Select different GC settings to use for different sites. Choose a Default to fall back to or now, as you please.
Somewhat Hard: Define templates and regular expressions for different GC settings in personal.ini. Allows for GC setting by category or genre (or author, or whatever) across different sites, or with different settings per site.


By leaving FFDL's basic 'Update Cover' option on and using GC's '{Current Cover}' feature, you can also get GC to use a cover image from the story automatically.

Here's an example of the personal.ini config:
## If you have the Generate Cover plugin installed, you can use the
## generate_cover_settings parameter to intelligently decide which GC
## setting to run. There are three parts 1) a template of which
## metadata part(s) to look at, 2) a regular expression to match the
## template, and 3) the name of the GC setting to use, which must
## match exactly.
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
## template => regexp to match => GC Setting to use.
generate_cover_settings:
${category} => Buffy:? [tT]he Vampire Slayer => BuffyCover
${category} => Star Trek => StarTrekCover

(Trying to do a GUI version of generate_cover_settings is a lot of work that I'm only going to invest if there's a lot of interest.)

(zip file removed after official release)

Aleyst
05-03-2012, 12:18 AM
OOH! *happy dance* I'm so testing this!
Cheers!

---
Tested on FF.net stories, works a treat using Category.
Any way to get it to work via the "Character" part?

JimmXinu
05-08-2012, 11:52 AM
v1.5.18 posted with new feature:
(1.5.17 skipped)

Automatically call Generate Cover(GC) on FFDL download/update.

There's a new config tab in FFDL for Generate Cover and three ways to set it up:

Easiest: Select GC setting for 'Default' to use for all downloads/updates.
Still Easy: Select different GC settings to use for different sites. Choose a Default to fall back to or now, as you please.
Somewhat Hard: Define templates and regular expressions for different GC settings in personal.ini. Allows for GC setting by category or genre (or author, or whatever) across different sites, or with different settings per site.


By leaving FFDL's basic 'Update Cover' option on and using GC's '{Current Cover}' feature, you can also get GC to use a cover image from the story automatically.

Here's an example of the personal.ini config:
## If you have the Generate Cover plugin installed, you can use the
## generate_cover_settings parameter to intelligently decide which GC
## setting to run. There are three parts 1) a template of which
## metadata part(s) to look at, 2) a regular expression to match the
## template, and 3) the name of the GC setting to use, which must
## match exactly. Use this parameter in [defaults], or by site eg,
## [www.ficwad.com]
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
## template => regexp to match => GC Setting to use.
generate_cover_settings:
${category} => Buffy:? [tT]he Vampire Slayer => BuffyCover
${category} => Star Trek => StarTrekCover

(Trying to do a GUI version of generate_cover_settings is a lot of work that I'm only going to invest if there's a lot of interest.)

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

cklls
05-09-2012, 12:49 PM
Thank you once again! Love the new Cover options. :)

JimmXinu
05-09-2012, 02:44 PM
Testing support for a new site:

"The WWOMB" http://www.squidge.org/peja/cgi-bin/index.php

This is a bit different than others--there's several unrelated areas under the domain squidge.org. This is support only for stories in the eFiction instance under squidge.org/peja.

This necessitated some changes to the internals and will require more if we ever want to add support for additional sites under squidge.org.

Let me know what you think.

anesthezea
05-09-2012, 04:59 PM
I got an email this morning from the mod of the 852 Prospect Archive (I'm a member of the TS mailing list) and it seems there are plans in place to move the entire archive over to Archive of Our Own. I'm not sure when this will take place, but I'm looking forward to it! :)

I thought I'd pass word along since 852 is on your list of possible sites to include in future.

JimmXinu
05-09-2012, 05:42 PM
Thanks! I've made a notation to the list.

bb63
05-11-2012, 08:45 PM
hi all. i'm very new to calibre and to the plug in concept!

love all the work you guys do here!!!

are there any plans to add live journal to this?

JimmXinu
05-11-2012, 08:55 PM
@bb63 - Glad you like it.

As for live journal, nope. From what I've seen it really isn't meant to be a story system. But if you can give me several URLs to live journal stories, I might reconsider it.

There are other tools available. Another user said: "...Squeebook.net is an online tool that grabs/formats stories from livejournal/dreamwidth journals." The GrabMyBooks FireFox plugin is another possibility.

Firedancer885
05-12-2012, 03:43 AM
Hi,

I just found the plugin but fell in love with it already. I have a support question/request.
I downloaded a great deal of stories from fanfiction net via the deffnetizer program or the ficsavers.com website. Both of those put the story url at the beginning of the story text in id:xxxxxxx link format. Is there a way to make the "get URLs from selected books" option recognize those links as "url:" identifiers? I don't want to troll through all of my nearly 150 epubs for those urls.

JimmXinu
05-12-2012, 11:41 AM
@Firedancer885 - Hmm. Not a bad idea. Can you provide a (preferably smallish) example of each?

Firedancer885
05-12-2012, 03:57 PM
@Firedancer885 - Hmm. Not a bad idea. Can you provide a (preferably smallish) example of each?

Ok, I've run into a snag. I thought the ficsavers.com site uses the same template as the deffnetizer program but it doesn't. The ficsavers.com template uses a botched up url :( It's "story name" by "author", and the story name is the link to the story. Only some slashes are missing along with the chapter number :(

Harrowed uses the id:xxxxxxx template, The Leap uses "story name" by "author". I'd be thankful for any help.

ps: And then I realize not all of the ficsavers.com stories have botched up urls. :( Sorry :(

JimmXinu
05-12-2012, 04:53 PM
I've looked at deffnetizer and ficsavers.com a bit. Neither outputs epub directly, so apparently you've already converted from html to epub.

The deffnetizer epub you posted does contain a proper story URL, but that depends on the output template used. The first two templates I tried, "Default Template" and "Minimal" do not include the story URL anywhere.

The ficsaver.com epub you posted, as you say, contains a broken version of the story URL. Although it looks like that may only be a problem with ffnet stories.

All told:

The user would have to have converted the story to epub;
Then the plugin would have to scan at least the first three html files (cover page, ToC/splash page and first real page) for URLs, while;
Checking for both valid story URLs and at least one form of broken URL;
And even then, it wouldn't work for half of the included deffnetizer output templates.

So, while possible, it would be a bit of a pain and not work all the time anyway. At this point, I'll add it to the 'wishlist' of features I might add some day, but it's not going to be a priority.