[GUI Plugin] FanFicFare

[Tanjamuse]

The only place I can see the characters at all is at the search site where there is a category which enables me to search for stories based on the one character on the list.

When I download a story even though I can find it based on a character it doesn't show those characters anywhere in the downloaded story's metadata (not even in the metadata-column).

I had thought that I could use the search function on the site and add the character in the search to the characters column I have in my metadata, and it worked fine until I stumbled across a story to which I've already added a character. It then deleted the existing metadata. This only happens when I select multiple stories.

[dennocoil]

After much trial and error, I got the concept of how it's supposed to work and it works now. Just to get some clarification, what does this have to do in relation to everything else?

extra_valid_entries:likes
likes_label:Likes

[JimmXinu]

Originally Posted by dennocoil

After much trial and error, I got the concept of how it's supposed to work and it works now. Just to get some clarification, what does this have to do in relation to everything else?

extra_valid_entries:likes
likes_label:Likes

What I call "standard" metadata entries are hard coded and common to most adapters. Standard list entries Standard scalar entries

Many site adapters have what I call "site specific" metadata entries that they collect. The system is informed about these using extra_valid_entries so it knows to add them to the list (they are added as 'list' entries rather than scalar). Only valid entries can be included in title pages, used in replace_metadata, added to extra_subject_tags, etc.

An individual user can also add their own for even more sophisticated processing.

Any valid metadata entry, either standard or 'extra' can be given a label with <entryname>_label that will be used in the titlepage and logpage. If <entryname>_label is not present, Python's .title() function will be applied to the entry name. See code.

[Tanjamuse]

I've found a solution to my problem with adding new characters. It may not be the most elegant, but it's working.

[cryzed]

@JimmXinu, is there a way of backing up old versions of a story when updating it (fully)? Sometimes authors or sites decide that they want to remove an entire story or delete the chapter contents -- updating a story then will effectively delete the story from my library; I would like to prevent that.

If there's no obvious way to do it that I'm simply overlooking, I might try my hand at adding such an option.

[Tanjamuse]

That would be a good idea.

[cryzed]

Hm, it's kind of hard to do since the file writing/updating for Calibre jobs seemingly works in a pretty complicated fashion -- workers get a target path that is supposed to be written to, but from that I don't think I can easily recreate the final path, so a custom backup path would have to be specified. However, even then, the file behind the outfilename that arrives in BaseStoryWriter.write_story() seems to be empty when the story is fully updated -- so one would have to change the code in all places where this method is called (3 places for the plugin).

Maybe Jimm can chime in with the best approach, I'm probably overlooking something obvious since I'm largely unfamiliar with the Calibre Plugin API.

[cryzed]

This is what I have so far (it's a fixed backup destination currently, which has to be changed). Unfortunately the resulting backup files are always empty -- which confuses me, because writeStory() uses the data presumably contained within to check if the EPUB should be updated in the first place? Or maybe it's only applicable to the CLI version and Calibre's Plugin API always creates an empty target file?

[edeniz]

Originally Posted by cryzed

@JimmXinu, is there a way of backing up old versions of a story when updating it (fully)? Sometimes authors or sites decide that they want to remove an entire story or delete the chapter contents -- updating a story then will effectively delete the story from my library; I would like to prevent that.

If there's no obvious way to do it that I'm simply overlooking, I might try my hand at adding such an option.

If I may butt in, cryzed:

In my experience, when a story gets deleted or only some chapters get deleted, calibre plugin will skip the update automatically with the message that either the story does not exist or the story you're trying to download has fewer chapters than the file you already have. The sole requirement for that to happen is that you choose the option "update epub if new chapters" instead of "overwrite if newer" option.

[cryzed]

Hey edeniz, you aren't wrong, however that isn't a fool-proof solution. Regularly I overwrite the entire story, not just update it, since I read quite a few on sites where it is common to make changes to older chapters (rewrites etc.), and I'd like to keep them for the archive.

[davidfor]

Originally Posted by cryzed

Hm, it's kind of hard to do since the file writing/updating for Calibre jobs seemingly works in a pretty complicated fashion -- workers get a target path that is supposed to be written to, but from that I don't think I can easily recreate the final path, so a custom backup path would have to be specified. However, even then, the file behind the outfilename that arrives in BaseStoryWriter.write_story() seems to be empty when the story is fully updated -- so one would have to change the code in all places where this method is called (3 places for the plugin).

Maybe Jimm can chime in with the best approach, I'm probably overlooking something obvious since I'm largely unfamiliar with the Calibre Plugin API.

That to me says you are taking the backup at the wrong time. The chapter gathering has already determined what has changed. From that, you should be able to do the backup before the download starts and access the original file.

But, calibre already has a backup method built in. It is probably possible to trigger the creation of the ORIGINAL_xxx that can be created when doing a conversion. Doing that has the advantage that calibre is expecting these files. And it is easy to restore them or use the editor to compare them.

And, the code to do this doesn't look hard. It is in gui2/tools.py, method convert_single_ebook. That does need access to the database, so it probably needs to be done before the download job is started.

[edeniz]

Originally Posted by cryzed

Hey edeniz, you aren't wrong, however that isn't a fool-proof solution. Regularly I overwrite the entire story, not just update it, since I read quite a few on sites where it is common to make changes to older chapters (rewrites etc.), and I'd like to keep them for the archive.

Okay, thanks for the explanation, cryzed. Looking at things from that angle, then of course a rewrite makes sense, as well as wanting to find a way to protect the previous versions. I was confused and did not quite understand what you needed. Hope you can find a good solution, then.

[JimmXinu]

New test version posted

2017-07-02
- Fix for XenForo(SB/SV) threadmarks with non-ascii chars. (web & CLI)

FYI, I have company for a few days, so my free time is a bit limited right now.

Re: Built-in backup, as edeniz said, the first line of defense is 'update epub if new chapters'. The second line of defense, IMO, is your own regular backups--everybody has those, right?

ORIGINAL_xxx as davidfor suggested is a possibility, but there are other things that also use that. So you might unknowingly overwrite your backup--maybe even automatically with auto-Convert. Another issue is that if you use certain tools like Polish on 'EPUB' and you also have an ORIGINAL_EPUB file, it will use the 'original' instead of the EPUB.

Copying the pre-update epub(assumed) to a different dir outside calibre entirely is possible, but not very elegant. Look at db.copy_format_to() (fff_plugin.py:1366) and add a second call there to copy to your backup dir (after you've figured out a file name, of course).

Other possibilities are to copy pre-update to another library, or maybe even define a new format extension to copy a backup to. (That would be in conflict with the 'Delete other formats' feature.)

I believe 'backups' have been requested before and I rejected the idea as outside scope and a bit of a slippery slope. If one backup, why not the last 2? Or 3? What about comparing to show what's changed between backup and current? Etc, etc.

[davidfor]

Originally Posted by JimmXinu

Re: Built-in backup, as edeniz said, the first line of defense is 'update epub if new chapters'. The second line of defense, IMO, is your own regular backups--everybody has those, right?

ORIGINAL_xxx as davidfor suggested is a possibility, but there are other things that also use that. So you might unknowingly overwrite your backup--maybe even automatically with auto-Convert. Another issue is that if you use certain tools like Polish on 'EPUB' and you also have an ORIGINAL_EPUB file, it will use the 'original' instead of the EPUB.

Copying the pre-update epub(assumed) to a different dir outside calibre entirely is possible, but not very elegant. Look at db.copy_format_to() (fff_plugin.py:1366) and add a second call there to copy to your backup dir (after you've figured out a file name, of course).

Other possibilities are to copy pre-update to another library, or maybe even define a new format extension to copy a backup to. (That would be in conflict with the 'Delete other formats' feature.)

I believe 'backups' have been requested before and I rejected the idea as outside scope and a bit of a slippery slope. If one backup, why not the last 2? Or 3? What about comparing to show what's changed between backup and current? Etc, etc.

Yes, the backup is a slippery slope. With the ORIGINAL_xxx, I wasn't thinking of the other functions that might use this. I think I saw the backup as a safety feature and remove once you have checked the update is OK. And I'd forgotten about the Polish issues as the reason I knew where the code for this was was because I already had been fiddling with it to prevent use of the ORIGINAL_xxx in certain circumstances. (If I have an epub and an ORIGINAL_epub and do a conversion to kepub, which am I more likely to want to use?)

Of all the other choices, I'd be most tempted by adding another format such as "FFFBackup_xxx". The hassle with that is the thing I think I would most want to use it for is a comparison to the updated version. That would require a lot more changes than would be worthwhile.

[madame_newbie]

Is the a way for me to bypass the 'Change Story URL' pop up. I am currently updating my identifiers and it's getting very tedious. I have looked in the .ini settings and can't find one that sets the answer to 'yes' automatically.