[GUI Plugin] FanFicFare

[dailyalice]

If anyone familiar with the different date options in calibre & FanFicFare has the time and inclination to read this over and see if it makes sense and is accurate, I'd very much appreciate it. Want to get this right.

The paragraphs after this one are what I came up with when trying to document/explain the different date column options with calibre & FanFicFare. (I may yet simplify the language. I tend to be wordy.) I go over the calibre default date columns first, then get to FanFicFare. I fear it is clear as mud.

The column with the column header of "Date" and the lookup name of "timestamp" is literally for when you added a work to your calibre library. I find this handy if I want to sort my library based on the items I've added to it most recently. It appears to use the default format of dd MMM yyyy (which would give you dates like 28 Sep 2022).

The column with the column header of "Published" and lookup name of "pubdate" is used for the date a work was published; it appears to only include the month and the year (MMM yyyy); if you want something more specific than that which includes the day of the month, you'll want to create a column for FanFicFare to use. These dates look like "Sep 2022" in calibre.

The column with the column header of "Modified" and "Lookup name" of "last_modified" tells you when you last edited a work or made any changes to it. It also uses the default format of dd MMM yyyy. I just tested this by editing the metadata of a work to remove a single tag; now the "Modified" date is for today's date.

These calibre standard columns cover Date (you added a work to your library), Published (month/year a work was published/posted), and Modified (when you last made changes to a work either by hand or using FanFicFare). Date and Modified use the format dd MMM yyyy; Published uses MMM yyyy.

What about FanFicFare? The items which can be put into columns with a date column type are: Published, Updated, & Created. If you create custom columns to use with this data, you can format the dates however you like.

"Published" is the publication date, which actually does include the month, day, & year info (as opposed to calibre's Published column which just has month & year).

"Updated" is when a work was last updated (by the author on the archive from which you downloaded it) and it also includes day, month, year.

"Created" is usually labeled "Packaged" by FanFicFare (this is defined in personal.ini & can be changed) and is the date (& time) when you downloaded a work (using FanFicFare).

And there's also the special column "FFF Updated" which covers "the last time FanFicFare updated or checked for updates" which is different from when you last edited anything having to do with a work or when a work was updated.

[JimmXinu]

This is mostly good, but there are some additional nuances.

How timestamp, pubdate and last_modified appear can be customized in Calibre Preferences > Tweaks > "Control how dates are displayed". I suspect that the default appearances are based on locale, but I could be wrong.

All three are internally stored with time as well as date. So tweaking the appearance may reveal previous stored, but hidden timestamp precision.

Originally Posted by dailyalice

The column with the column header of "Date" and the lookup name of "timestamp" is literally for when you added a work to your calibre library. I find this handy if I want to sort my library based on the items I've added to it most recently. It appears to use the default format of dd MMM yyyy (which would give you dates like 28 Sep 2022).

That is how Calibre treats timestamp. FFF by default updates timestamp when ever a downloaded book is updated. This can be changed by checking FFF's Config > Standard Columns > Date New Only.

Originally Posted by dailyalice

The column with the column header of "Published" and lookup name of "pubdate" is used for the date a work was published; it appears to only include the month and the year (MMM yyyy); if you want something more specific than that which includes the day of the month, you'll want to create a column for FanFicFare to use. These dates look like "Sep 2022" in calibre.

pubdate can be tweaked to show more precision, no added custom column needed. The column does record time as well as date. Not all FFF supported sites have pubdate with time, but for those that do, it is set in pubdate.

Originally Posted by dailyalice

The column with the column header of "Modified" and "Lookup name" of "last_modified" tells you when you last edited a work or made any changes to it. It also uses the default format of dd MMM yyyy. I just tested this by editing the metadata of a work to remove a single tag; now the "Modified" date is for today's date.

I would note that last_modified can change unexpectedly sometimes.

For example, if you have FFF downloaded books with the same author name but from different sites, downloading one may change the author URL for that author name. When that happens, all books with the same author name (even from different sites), will get the new author URL and last_modified reset. Or if a tag is changed in the tag browser, all books with that tag will have last_modified reset.

This is something that surprises Calibre users in general fairly regularly.

Originally Posted by dailyalice

What about FanFicFare? The items which can be put into custom columns with a date column type are: Published, Updated, & Created. If you create custom columns to use with this data, you can format the dates however you like.

I would emphasize the difference between standard and custom columns more, including the fact custom lookup names all start with # and that the user must create them.

Originally Posted by dailyalice

"Published" is the publication date, which actually does include the month, day, & year info (as opposed to calibre's Published column which just has month & year).

Again, tweaks for Calibre pubdate. And some sites have time, too.

Originally Posted by dailyalice

"Updated" is when a work was last updated (by the author on the archive from which you downloaded it) and it also includes day, month, year.

"Created" is usually labeled "Packaged" by FanFicFare (this is defined in personal.ini & can be changed) and is the date (& time) when you downloaded a work (using FanFicFare).

Updated in FFF also has time on some sites. Easiest way to know which sites is to search in defaults.ini for _format. Created/Packaged always has time.

Originally Posted by dailyalice

And there's also the special column "FFF Updated" which covers "the last time FanFicFare updated or checked for updates" which is different from when you last edited anything having to do with a work or when a work was updated.

I wouldn't describe this has a 'special column', but as a FFF custom column setting. And it's labeled 'Last Checked Column' under FFF Config > Custom Columns. I assume 'FFF Updated' is your name for the column.

[applegaa]

Originally Posted by dailyalice

I've been assuming most (all?) sites give a status of Completed or In-Progress or that FanFicFare interprets things that way, but wanted to make sure I had this right. Are there other statuses supported for other sites?

Here is my current list of statuses:

In-Progress,
Completed,
Hiatus,
Abandoned,
Dropped,
Stub,
Deleted,
Vacation,
Anthology

I leave the column blank if the site does not return a status.

I added the vacation and anthology options because I set that status manually when appropriate. I haven't seen a site that uses them. I treat vacation like hiatus and check less frequently for updates.

- Andrew

[ownedbycats]

Originally Posted by applegaa

Here is my current list of statuses:

In-Progress,
Completed,
Hiatus,
Abandoned,
Dropped,
Stub,
Deleted,
Vacation,
Anthology

I leave the column blank if the site does not return a status.

I added the vacation and anthology options because I set that status manually when appropriate. I haven't seen a site that uses them. I treat vacation like hiatus and check less frequently for updates.

- Andrew

I have
- In-Progress
- Complete
- Oneshots
- Dormant
- Abandoned
- Anthology
- Fanzines

"Oneshot" is completed fics with one chapter. Dormant means it hasn't been updated in a long time. Abandoned is usually when it's deleted, but also a few where the author died () or has stated they've abandoned it.

[dailyalice]

Thanks so much for this thorough reply, Jim!

Originally Posted by JimmXinu

This is mostly good, but there are some additional nuances.

How timestamp, pubdate and last_modified appear can be customized in Calibre Preferences > Tweaks > "Control how dates are displayed". I suspect that the default appearances are based on locale, but I could be wrong.

All three are internally stored with time as well as date. So tweaking the appearance may reveal previous stored, but hidden timestamp precision.

This is very good to know. I confess that after getting back to using calibre after not using it in years, I haven't thoroughly explored all the features and have forgotten about some of them (and some are new since I last used calibre a lot). I hadn't even looked at Tweaks this time around.

Though I went to see about adding day of week to the publication date format and the "Apply" button in tweaks stayed greyed out even after I edited it. Tried editing other Tweaks and had the same issue. Maybe it's a known bug in calibre 6.8.0? Or something is awry on my system that I can't apply/change any Tweaks. Weird.

Originally Posted by JimmXinu

That is how Calibre treats timestamp. FFF by default updates timestamp when ever a downloaded book is updated. This can be changed by checking FFF's Config > Standard Columns > Date New Only.

Got it.

Originally Posted by JimmXinu

pubdate can be tweaked to show more precision, no added custom column needed. The column does record time as well as date. Not all FFF supported sites have pubdate with time, but for those that do, it is set in pubdate.

Okay, so there's really no reason to add an additional custom column for publication date as it looks like you can even add time to the pubdate format in Tweaks.

Originally Posted by JimmXinu

I would note that last_modified can change unexpectedly sometimes.

For example, if you have FFF downloaded books with the same author name but from different sites, downloading one may change the author URL for that author name. When that happens, all books with the same author name (even from different sites), will get the new author URL and last_modified reset. Or if a tag is changed in the tag browser, all books with that tag will have last_modified reset.

This is something that surprises Calibre users in general fairly regularly.

Oh, this is an excellent point and something I haven't really covered elsewhere in the guide (so far). And I can see how people would easily miss it.

If you change any sort of tag that is used for multiple works (including author names), those works are considered changed/updated. It makes sense! But I can see how folks might miss that. And, as you pointed out, the same holds if FanFicFare does it.

Originally Posted by JimmXinu

I would emphasize the difference between standard and custom columns more, including the fact custom lookup names all start with # and that the user must create them.

I have that covered pretty well elsewhere in the guide, but appreciate you mentioning it (in case I hadn't had that covered). I should probably make it standard within the guide to use the # in front of custom column names when referring to them by column lookup name.

Originally Posted by JimmXinu

Again, tweaks for Calibre pubdate. And some sites have time, too.



Updated in FFF also has time on some sites. Easiest way to know which sites is to search in defaults.ini for _format. Created/Packaged always has time.

Noted. I'll add the time info to the relevant paragraphs.

Originally Posted by JimmXinu

I wouldn't describe this has a 'special column', but as a FFF custom column setting. And it's labeled 'Last Checked Column' under FFF Config > Custom Columns. I assume 'FFF Updated' is your name for the column.

I refer to the three FanFicFare columns listed under "Special column" in the "Custom Columns" tab of FFF Config as "special columns" throughout the guide. They're custom columns meant for use with special FanFicFare specific data, I guess? This is all in a section about custom columns. See attached for column names etc.

For consistency I probably should have named "Saved Metadata" with "FFF" at the start of it too, but I figure there's no confusing that custom column for any others I might create whereas there could conceivably be for Updated. Probably not with Errors.

[JimmXinu]

Originally Posted by dailyalice

Though I went to see about adding day of week to the publication date format and the "Apply" button in tweaks stayed greyed out even after I edited it. Tried editing other Tweaks and had the same issue. Maybe it's a known bug in calibre 6.8.0? Or something is awry on my system that I can't apply/change any Tweaks. Weird.

Day of week isn't one of the offered parts, so I'm not hugely surprised it doesn't work. (Day of week? Why??)

Originally Posted by dailyalice

I refer to the three FanFicFare columns listed under "Special column" in the "Custom Columns" tab of FFF Config as "special columns" throughout the guide. They're custom columns meant for use with special FanFicFare specific data, I guess? This is all in a section about custom columns. See attached for column names etc.

Fair enough. I was reading in the context of coming right after paragraphs about standard columns.

I'm excited to see someone documenting things so well.

[dailyalice]

Originally Posted by JimmXinu

Day of week isn't one of the offered parts, so I'm not hugely surprised it doesn't work. (Day of week? Why??)

I totally misspoke there, I meant day in addition to month and year. Was trying to add that to pubdate. Day of the week would be weird and quite pointless.

[dailyalice]

Originally Posted by applegaa

Here is my current list of statuses:

In-Progress,
Completed,
Hiatus,
Abandoned,
Dropped,
Stub,
Deleted,
Vacation,
Anthology

I leave the column blank if the site does not return a status.

I added the vacation and anthology options because I set that status manually when appropriate. I haven't seen a site that uses them. I treat vacation like hiatus and check less frequently for updates.

Interesting. Thanks for sharing these!

Originally Posted by ownedbycats

I have
- In-Progress
- Complete
- Oneshots
- Dormant
- Abandoned
- Anthology
- Fanzines

"Oneshot" is completed fics with one chapter. Dormant means it hasn't been updated in a long time. Abandoned is usually when it's deleted, but also a few where the author died () or has stated they've abandoned it.

Also interesting, I'm fascinated by how people handle this stuff.

Somewhere else people were asking others what amount of time has to pass until someone considers a work "abandoned" and people definitely had different opinions. For me it isn't necessarily about the time period, I've seen too many authors resume a work years later (always so pleased when that happens), but about whether they seem to be still active at all I guess.

It never occurred to me to put "Anthology" in "Status" so I'm surprised to see that you both do that. Huh. I guess most of the anthologies I'm building these days are created from series on AO3 and AO3 has completion status for series and I use that.

(Are you using column types where you can have multiple options listed for status? As I mentioned earlier, I was previously using a fixed set of values. Was going to switch to a text column but most of the time I use tag columns and now I'm pondering if there are cases where I'd want multiple status items for a work.)

[JimmXinu]

Personally, I use a few manually set variants of Abandoned: Abandoned(replaced), Abandoned(removed), Abandoned(Author deceased). Sadly, that last keeps growing.

But I just put them in Tags. The auto complete is good enough for me for keeping consistent tags.

[ownedbycats]

Originally Posted by dailyalice

Somewhere else people were asking others what amount of time has to pass until someone considers a work "abandoned" and people definitely had different opinions. For me it isn't necessarily about the time period, I've seen too many authors resume a work years later (always so pleased when that happens), but about whether they seem to be still active at all I guess.

I like to check my 'dormant' fics from time to time (manually in the browser! Jim's warned enough about bulk-updates). Once I've been pleasantly surprised with the author beginning updates after a half-decade hiatus. Putting them in their own category away from abandoned fics helps with that.

EDIT: Also subscribing to the dormant fics so that if they do someday get an update I'll find out—either email, or in the case of ffnet's broken email updates via the story alerts page.

[JimmXinu]

New Alpha Test Version Attached

2022-11-13
- Update metadata caching with dependency invalidating

This version rewrites the way metadata caching works to track dependencies and invalidated derived values when a metadata entry is updated. It potentially sacrifices speed for improved accuracy in corner cases such as found earlier with cover_image.

It's been working correctly for me for several days, but my metadata setup is pretty modest and it does matter most with complex setups.

I am declaring this one an 'alpha' test--I judge that it is more likely to have problems than the usual test versions.

I invite users who understand that the metadata handling needs testing, and are willing to, to try it. Remember your backups.

I'm interested in both whether it's accurate and whether it sacrifices too much speed.

[ownedbycats]

I just tested it.

Cleared the #fanficcover field and ran an 'overwrite always' on a fic that had a cover image in the first chapter (do you need the link?)

It did populate #fanficcover, but with "DownloadedDownloaded." I'm not sure if that's a mistake in my config (unchanged since the last I posted) or somewhere else (something unintentionally duplicating the values?). edit: as mentioned in post below, regex issue.

If any use, the cover image bit in #fanficmetadata read like this:

<p><span class='label'>Cover_Image</span>: <div class='metadata' id='cover_image'>first</div><p>

I noticed it took maybe a few seconds longer to update than I usually expected but nothing too much unreasonable.



I also got a DownloadedDownloaded when doing "update if new chapters" on a fic with no cover image (generated cover). I think this is on account of the 'old' cover_image.

This shouldn't be too much of a problem in practice. Either I edit the regex to exclude 'old,' and/or use custom_columns_settings to apply only to new books.

edit: further tests below

[ownedbycats]

Test #2:

I commented out covertype=>(.*)=>Downloaded. This time I got just 'old.'

So I tried

 covertype=>old=>Downloaded

on a fic with generated cover and got just 'Downloaded.' So, regex issue.

Test #3: Got it!

 covertype=>(specific|first|default|force)=>Downloaded

I got 'downloaded' for fics with cover-images, and it left a pre-existing 'Generated' alone!

Still curious in why (.*) would result in it doubling though, lol.

[JimmXinu]

Something with python3 changed regex somehow and * isn't as greedy as it used to be, I think.

Use ^.*$ to match an entire string now.

[ownedbycats]

## If set, force_cover_image will be made the cover, taking precedence
## over specific story cover, first image, default and
## pre-existing(aka old) EPUB cover image.
## It can be either a 'file:' or 'http:' url.
## force_cover_image is a python string Template string with
## ${title}, ${author} etc, same as titlepage_entries. Unless
## allow_unsafe_filename is true, invalid filename chars will be
## removed from metadata fields
#force_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
#force_cover_image:file:///C:/Users/username/Desktop/nook/images/${title}/icon.png
#force_cover_image:http://www.somesite.com/someimage.gif

How do I handle spaces in a file:// path, specifically the folder name (e.g. D:/Documents/My Writings/)? It never actually occurred to me before because I usually use file:///D:/Pictures/Fanfics/ or a URL.

It doesn't seem to like %20s:

ERROR: Unhandled exception: <b>InterpolationSyntaxError</b>:'%' must be followed by '%' or '('

What should I do here?

EDIT: Replacing the %20s with actual spaces seems to work. Perhaps this could be mentioned in the default.ini for future people?