[GUI Plugin] FanFicFare

[theArchitect21]

Originally Posted by PandathePanda

Hi

try what is written here: https://www.mobileread.com/forums/sh...postcount=3308

Thank you!

[JimmXinu]

New Test Version Posted

2019-04-14
- More fixing for bool metadata values--convert to string when set.

Okay, now I think I've got it fixed.

[jesscat]

Yes that works perfectly now!

Now another question... Is there anywhere that the bookmark features for AO3 series bookmarks is retained, since AO3 permits bookmarking of series as well as individual stories? Sometimes I don't bookmark all x stories in a series, but just the series, so it would be nice if I could use this to populate my #my_bookmarks column for anthologies.

Or maybe if not (I'm thinking maybe it's not possible if AO3 is the only archive that has these features?) perhaps is there a way to use the bookmarked status of one of the stories in the anthology to trigger a "bookmarked" status for the entire anthology?

I can't believe the comprehensiveness of this plug-in and the amazing things that can be done with it and how responsive (and patient) you are - truly a fanfic reader's fantasy!

[jesscat]

Ugh, sorry to spam with questions, but I can't find an answer to this one either, despite searching extensively...

I really love the logpage feature in the epubs, but there doesn't seem to be a lot about it in this thread. I've modeled mine after your example, and it works great. But what I'd like to include in my update log is the actual text in the "Comment" field of the log that calibre provides when it does an update and then lets me view the log before making the changes to my library - something like:

Update epub completed, added 1 chapters for 10 total.

Is there a metadata field or other way to access this? I don't see one in default.ini.

[JimmXinu]

Originally Posted by jesscat

Yes that works perfectly now!

Glad that's fixed it.

Originally Posted by jesscat

Now another question... Is there anywhere that the bookmark features for AO3 series bookmarks is retained, since AO3 permits bookmarking of series as well as individual stories? Sometimes I don't bookmark all x stories in a series, but just the series, so it would be nice if I could use this to populate my #my_bookmarks column for anthologies.

Or maybe if not (I'm thinking maybe it's not possible if AO3 is the only archive that has these features?) perhaps is there a way to use the bookmarked status of one of the stories in the anthology to trigger a "bookmarked" status for the entire anthology?

Bookmarks on works are on a page we already load. Detecting series bookmarks would require another page we don't already load. I'd prefer not to add that overhead unless there's more than one user wants it--it's a bit esoteric.

Series URLs for anthology epubs are loaded using the generic 'Get Story URLs from Web Page' feature which doesn't parse for site-specific info.

Originally Posted by jesscat

I really love the logpage feature in the epubs, but there doesn't seem to be a lot about it in this thread. I've modeled mine after your example, and it works great. But what I'd like to include in my update log is the actual text in the "Comment" field of the log that calibre provides when it does an update and then lets me view the log before making the changes to my library - something like:
...
Is there a metadata field or other way to access this? I don't see one in default.ini.

The logpage feature didn't get a lot of discussion because, until now--to the best of my knowledge--nobody but me used it.

The download 'comment' isn't metadata collected about the book. In fact, it isn't generated until after the output file has been written. Because if FFF failed to write the output file, that's where it would report it.

If you're interested, here's a snippet from my personal.ini. I use an extra entry to explicitly mark when an epub has been overwritten, plus there's words_added which... may not be documented anywhere.

Spoiler:

[epub]
include_tocpage: false
include_titlepage: true
include_logpage: smart
logpage_entries: fulldownload,title,storyUrl,series,author,category,genre,rating,warnings,description,dateCreated,datePublished,dateUpdated,numChapters,numWords,words_added,status,version
keep_summary_html:true
make_firstimage_cover:true

# so overwrite downloads have some marker.  
add_to_extra_valid_entries:,fulldownload
include_in_fulldownload:title
fulldownload_label:New Download

add_to_replace_metadata:
 fulldownload=>^.*$=>Yes

[jesscat]

Bookmarks on works are on a page we already load. Detecting series bookmarks would require another page we don't already load. I'd prefer not to add that overhead unless there's more than one user wants it--it's a bit esoteric.

Fair enough. Maybe someone else will express interest at some point!

Series URLs for anthology epubs are loaded using the generic 'Get Story URLs from Web Page' feature which doesn't parse for site-specific info.

I'm not sure I completely understand this? Do you mean that when anthologies are created they don't gather site-specific fields for each story in the URL list, like "bookmarked," which is AO3-specific? So there wouldn't be a way to assign the anthology a "bookmarked" status based on the status of the individual members? Or is this still referring to the AO3 series page information?

If you're interested, here's a snippet from my personal.ini. I use an extra entry to explicitly mark when an epub has been overwritten, plus there's words_added which... may not be documented anywhere.

I had found a version of that snippet and had been using it, but it did not include the words_added! I am going to add that right now!

The download 'comment' isn't metadata collected about the book. In fact, it isn't generated until after the output file has been written. Because if FFF failed to write the output file, that's where it would report it.

Yes, I see. Still, it would be nice to be able to capture those log comments. Could they be recorded in a custom column maybe? I have a column set up to record update errors, but as far as I can tell, it doesn't record the update comments like that, which aren't errors. Though to make it really useful as a record, the column contents would have to be appended each time, not replaced...hmm.

I guess it doesn't seem that many other people are interested in this though!



Another series-related question that just occurred to me: is there a way that you can think of to put the title/series name of a series anthology into the "series" column with an designated index - probably of 0 just in case any of the composite stories were also downloaded individually (e.g., {series_name}[0])?

[jesscat]

Now I have an actual issue!

I tried the update from URL within email feature, using an update email I received from AO3, for a story that had already been downloaded into my library using FFF.

First I received an error message saying:

"An error has occurred while FanFicFare was updating calibre's metadata for [fic title]. The ebook has been updated, but the metadata has not."

calibre, version 3.40.1
ERROR: Could not open e-book: Failed to read book, D:\Jessica\Documents\Jessica's Stuff\My eBooks\Calibre Fanfic\crazy_like_a\The Unkindness of Ravens (365)\The Unkindness of Ravens - crazy_like_a.epub click "Show Details" for more information

Traceback (most recent call last):
File "site-packages\calibre\utils\ipc\simple_worker.py", line 289, in main
File "site-packages\calibre\ebooks\oeb\iterator\book.py", line 65, in extract_book
File "site-packages\calibre\customize\conversion.py", line 244, in __call__
File "site-packages\calibre\ebooks\conversion\plugins\epub_in put.py", line 267, in convert
File "site-packages\calibre\utils\localunzip.py", line 231, in extractall
File "site-packages\calibre\utils\localunzip.py", line 218, in _extractall
ValueError: Not a ZIP file

[JimmXinu]

Originally Posted by jesscat

Now I have an actual issue! ...

That error indicates that the conversion to another format failed--I assume you have auto-convert turned on.

I was able to download and update that story, with auto conversion to AZW3, without issue.

There's a chance that it really is a problem with the epub file, especially if you've been heavily customizing output options. Posting the offending epub somewhere and then PM'ing me a link would be useful. (mobileread.com doesn't like NSFW material posted.)

Do you get the same error if you try again? Or if you do an 'overwrite always' download? Can you view the epub file in Calibre, or does it give an error too?

If you do get the error, try running calibre in debug mode (under Calibre's Preferences dropdown) while you run the download, quit calibre and then post the debug log as txt attachment.

[JimmXinu]

New Test Version Posted

2019-04-15
- Add remove_authorfootnotes_on_update feature for AO3.

This is from a discussion on the google group about how AO3 authorfootnotes will end up at the end of multiple chapters when updating existing epubs.

## AO3 authorfootnotes and inspiredlinks end up in the 'last' chapter,
## but if updated, then there's a new 'last' chapter, leaving multiple
## copies.  This option applies the 'skip_on_ffdl_update' class to
## those tags which means they will be removed during epub reading for
## update.  This will only effect chapters added after turning the
## setting on.
## Result: Only the last chapter will have end notes.
## Side-effect: An 'Update Always' that doesn't add a new last
## chapter will also remove the end notes.
#remove_authorfootnotes_on_update:false

[JimmXinu]

Originally Posted by jesscat

I'm not sure I completely understand this? Do you mean that when anthologies are created they don't gather site-specific fields for each story in the URL list, like "bookmarked," which is AO3-specific? So there wouldn't be a way to assign the anthology a "bookmarked" status based on the status of the individual members? Or is this still referring to the AO3 series page information?

I was trying to explain why AO3 series bookmark cannot easily be collected.

While individual story bookmarked values are all collected together in the anthology's bookmarked value, it's a string list of the individual values, eg: "True,False,True" or even "True,True". That's not usable for setting a y/n custom column.

In theory, I could add code to accept a string list of true/false, but it's kinda ambiguous. "True,True"==True is fine, but what about "True,False,True" or "True,False"? For your purposes, you want any "True"=True, but other users may disagree...

Originally Posted by jesscat

Yes, I see. Still, it would be nice to be able to capture those log comments. Could they be recorded in a custom column maybe? I have a column set up to record update errors, but as far as I can tell, it doesn't record the update comments like that, which aren't errors. Though to make it really useful as a record, the column contents would have to be appended each time, not replaced...hmm.

What would that tell you that isn't already recorded better in logfile? I don't see enough utility in it justify a new optional feature for it.

Originally Posted by jesscat

Another series-related question that just occurred to me: is there a way that you can think of to put the title/series name of a series anthology into the "series" column with an designated index - probably of 0 just in case any of the composite stories were also downloaded individually (e.g., {series_name}[0])?

There's a feature "Check for existing Series Anthology books?" on the FFF config Basic tab to warn you if you're trying to download/update a story in a series you already have an anthology of.

Which admittedly doesn't help going the other way--adding a series anthology containing books you already have will not be flagged.

I could add an optional feature to put 'series 0' in an anthology's series, but having "series 0" in the anthology's series field isn't going to help all that much, either. It wouldn't appear until after you made the anthology, and you'd still have to check it manually.

[jesscat]

Originally Posted by JimmXinu

That error indicates that the conversion to another format failed--I assume you have auto-convert turned on.

I was able to download and update that story, with auto conversion to AZW3, without issue.

I do not have auto-convert turned on. I download in epub and read in epub. So I have no idea where that is coming from. But also, I probably should have included more information to my original post.

I got that error after trying my first update from email - a new chapter for an existing story that was already in my library and had downloaded fine the first time with FFF. At the time, I had FFF configured to:

- Download from email immediately
- Default to update metadata from saved metadata column

When I selected "Get Story URLs from Email," the update proceeded without asking for any further input from me - I assume because I had told it to download immediately. The update gave me the error:

"An error has occurred while FanFicFare was updating calibre's metadata for [fic title]. The ebook has been updated, but the metadata has not."

Then when I tried to open the ebook, I got the error I quoted already, that you said has something to do with converting, and couldn't open the book at all.

After thinking about it I wondered whether the problem arose from my default setting to update metadata from saved metadata column - maybe that screwed something up and it failed to update the chapter? (I assume updating from email would obey my defaults here right?) So I tried updating the same book again manually, with "Update Always," but I got the same error again.

Then I gave up, deleted the book, and re-downloaded it, which of course worked fine.

Unfortunately I did this all yesterday, before I read your reply, so I didn't run in debug mode etc., and now it's too late. I still feel like something went wrong that shouldn't have, but since no other stories in my library have updated with new chapters, and I have no emails with URLs in them, I can't see a way to try it again right now. If it arises again I'll attempt to gather debug information more carefully.

I've done very little customizing of the epub. Cover page (using the GC plugin), log page, and add_to_keep_html_attrs, that's about it.

[JimmXinu]

Originally Posted by jesscat

...
Then I gave up, deleted the book, and re-downloaded it, which of course worked fine.
...

If it happens again, save the epub first and try to get a debug log.

[JimmXinu]

New Version Released

Version 3.7.0 - 19 Apr 2019
- Revert "Remove defunct site www.destinysgateway.com"
- Recognize destinysgateway.com and www.destinysgateway.com
- Remove ncisfic.com -- moved to AO3.
- Fix date format for adapter_gluttonyfictioncom
- Comment out some old debugs.
- Add debug output for encoding used.
- Fix some comments.
- Add another StoryDoesNotExist string for adapter_fanfictionnet
- Closes #390 - RoyalRoad click link in emails.
- Update a comment link in setup.py.
- Include status 'Hiatus' for adapter_royalroadcom.
- Correct a comment in defaults.ini.
- Fix for saved custom column metadata and boolean values.
- Add bookmarked site specific metadata for adapter_archiveofourownorg.
- Fix for PI saved metadata not reading False & empty strings.
- More fixing for bool metadata values--convert to string when set.
- Add remove_authorfootnotes_on_update feature for AO3.
- Update Translations.

[jesscat]

I would like pairings separated with "&" on AO3 (eg "Harry & Severus") NOT to be included in "ships" (for purposes of my #ships custom column), but I'd still like them to be included in the Tags column.

I can easily remove the pairing from ships with replace_metadata. But I can't figure out how to do this while keeping the pairing in the tag list - if it isn't a separate tag or character, removing it from ships removes it from the tags. I thought about ships2chars, somehow putting this ship into Characters, but as far as I can tell I can't use that to accomplish what I want (though I have to admit I don't completely understand what ship2chars does and can't find much about it in default.ini or here in the forums, so I definitely could be wrong).

But I feel like there must be some way that's just out of my understanding. Do you have any suggestions?

Thanks!

[ilovejedd]

Originally Posted by jesscat

I would like pairings separated with "&" on AO3 (eg "Harry & Severus") NOT to be included in "ships" (for purposes of my #ships custom column), but I'd still like them to be included in the Tags column.

I can easily remove the pairing from ships with replace_metadata. But I can't figure out how to do this while keeping the pairing in the tag list - if it isn't a separate tag or character, removing it from ships removes it from the tags. I thought about ships2chars, somehow putting this ship into Characters, but as far as I can tell I can't use that to accomplish what I want (though I have to admit I don't completely understand what ship2chars does and can't find much about it in default.ini or here in the forums, so I definitely could be wrong).

But I feel like there must be some way that's just out of my understanding. Do you have any suggestions?

I believe ships2chars split the ship tags so in case you have a fic with ship tag Harry & Severus but doesn't have character Harry and character Severus, the separated character tags will be added.

Dunno how to do the above in personal.ini but removing platonic ship tags from a custom ship column is probably something easy to do with Calibre's bulk metadata editor as a saved search.

I think this might work.

Search
.+([&].+)+

Replace
(leave blank)