Modify ePub plugin dev thread

[chaley]

Originally Posted by kiwidude

Bearing in mind my complete ignorance on such dates and times, I tried adding these two lines to zipfile.py at line 1126:

                mtime = time.mktime(member.date_time + (0,0,0))
                os.utime(targetpath, (mtime, mtime))

This gets me "close", but every time I zip the file back up the hour increments by one. Or is that just an example of the timezone issue?

My guess is that you need to set the dst flag (the last zero) to -1.

[kovidgoyal]

To be robust I would suggest you create a time structure using time.localtime and then copy it changing the first six members and leaving the rest alone.

[JSWolf]

Originally Posted by jackie_w

Hi kiwidude,

I have a suggestion for your consideration re: the XPGT margins option.

The scenario is for a retail epub (minus DRM) which has a .xpgt file and has no need for a calibre epub-epub conversion. The epub will either have:
- margins in the .xpgt plus margins in the epub .css
- margins only in the .xpgt

For the former I can see that you would want to remove the .xpgt margins completely.

For the latter, if you remove the .xpgt margins, you would end up with no margins at all, which can be rather cramped on the device. In this case it might be nice to have the option to set .xpgt margins to match those specified in Prefs - Common Options - PageSetup. This should result in the same page margins you would have got if you'd done an epub-epub conversion, but without the conversion 'risk'.

There is a much better way to deal with the page template file. I do it all the time when there is one. I delete the file. No need for it. I remove the reference to it from the OPF file and I remove the reference to it from all the XML files. Then it is gone and gone forever. It was never needed in the first place. I've never see anyone do anything with it that you cannot do is CSS.

[kiwidude]

Originally Posted by capnm

Could you preserve the file dates (timestamps) when you unzip/rezip the epub?

Charles & Kovid have kindly put together a change to the zip functionality for this which will be in the next Calibre release.

[kiwidude]

Originally Posted by JSWolf

There is a much better way to deal with the page template file. I do it all the time when there is one. I delete the file. No need for it. I remove the reference to it from the OPF file and I remove the reference to it from all the XML files. Then it is gone and gone forever. It was never needed in the first place. I've never see anyone do anything with it that you cannot do is CSS.

I guess it depends on your definition of "better". I've applied the exact same approach that Calibre does in its conversions. This means only one file has to be changed, rather than having to change every single html file in the epub that links to it if you were to delete it completely.

I am sure you are 100% correct in that css can do everything an xpgt file does, but if that css is not yet present then only removing the margins from the xpgt is the "safest" approach.

Either way, it doesn't resolve the request from Jackie, who is effectively after an automation of applying a page setup to the xpgt file rather than removing it. Yes this "could" be done in css instead, but not easily automated.

[JSWolf]

Since I delete the page template file instead pf changing it, and I've not messed with it, can you achieve the same 0 margins you can via just CSS by making both the page template and CSS to 0 margins?

[jackie_w]

If it's too difficult (or contentious) put the idea aside for now. It's something I, personally, would use for some epubs because I don't like zero margins. However, I'm perfectly capable of hand-editing either the xpgt or css, or of doing the epub-epub conversion to fix the problem.

It's certainly not worth doing if I'd be the only person using it

[JSWolf]

Originally Posted by jackie_w

If it's too difficult (or contentious) put the idea aside for now. It's something I, personally, would use for some epubs because I don't like zero margins. However, I'm perfectly capable of hand-editing either the xpgt or css, or of doing the epub-epub conversion to fix the problem.

If you do an ePub > ePub conversion of say a publisher created ePub, you definitely can make it a bit of a mess. Calibre does a poor job sometimes going ePub > ePub. For example, if you have embedded fonts, Calibre takes the code out of the CSS and puts the @font code into every XML file. IT can also change the directory structure. So really, once the ePub is made, don't use Calibre to convert it. It's not pretty.

[drMerry]

2 more files I can think of:
thumbs.db
.DS_STORE

(I will have to look for the right capitals)
@artwork.
I did had an epub where I removed the artwork causing me to loose my cover (I did remove it manually, did not check the manifest file or other files, so you're right, everybody could do dumb-ass thinks with epub )

[drMerry]

Originally Posted by kovidgoyal

The innocence of youth. They just assume the time is in whatever the local timezone of the machine running the extractor is.

Look at it in this way. If timezone is not supported, it can not be set, so there is no way to bother about it. 1:00pm is 1:00pm if I create it in Japan, repack it in England (with same time-stamp) and look at it in Japan again, I still get 1:00pm back so you do not even have to think about the fact that different timezones exist

[kiwidude]

Originally Posted by drMerry

2 more files I can think of:
thumbs.db
.DS_STORE

(I will have to look for the right capitals)

These files (and iTunesArtwork for that matter) are already covered by the unmanifested files feature. I don't see the need to special case add functions for them.

@artwork.
I did had an epub where I removed the artwork causing me to loose my cover (I did remove it manually, did not check the manifest file or other files, so you're right, everybody could do dumb-ass thinks with epub )

Right, so you had one such ePub - what would be good to know is if there are any more like that and how it got into that state. Was it a one-off manual edit of the manifest by someone. Or could it have even been somone running a "add unmanifested files to manifest" type feature from epub-fix or some other tool on it at some point. I have no idea about the books that Apple sells via its store for instance - do they all apply such an approach to their books?

In the 0.2.1 beta that I put up I added some code to remove from the manifest. So at least its removal will be clean.

As for losing your "official" cover - that I am less concerned about. Particularly longer term when this plugin has the ability to replace covers in the ePub.

[drMerry]

Originally Posted by kiwidude

These files (and iTunesArtwork for that matter) are already covered by the unmanifested files feature. I don't see the need to special case add functions for them.

Just like your other remark, if it is included by a tool, it would not be unmanifested.

Originally Posted by kiwidude

Right, so you had one such ePub - what would be good to know is if there are any more like that and how it got into that state. Was it a one-off manual edit of the manifest by someone. Or could it have even been somone running a "add unmanifested files to manifest" type feature from epub-fix or some other tool on it at some point. I have no idea about the books that Apple sells via its store for instance - do they all apply such an approach to their books?

I have no Idea how I got that book. I am sure that the files where not added by add unmanifested option of this plugin. But earlier edits with other tools? No Idea.

Originally Posted by kiwidude

In the 0.2.1 beta that I put up I added some code to remove from the manifest. So at least its removal will be clean.

As for losing your "official" cover - that I am less concerned about. Particularly longer term when this plugin has the ability to replace covers in the ePub.

Well, If you only have this cover, you should concern I think.
So to safely remove this file (and any unmanifested file) you should not only check manifestation, but also check every txt-based file (htm, html, xhtml, xml, ...) to look for direct links to this file.

[capnm]

Originally Posted by kiwidude

Charles & Kovid have kindly put together a change to the zip functionality for this which will be in the next Calibre release.

Sweet!
I can see that pretty soon I'm going to run out of excuses for not cleaning up my libraries

Then it's going to be hours of OCD fussing and very little reading .....

[capnm]

Originally Posted by drMerry

2 more files I can think of:
thumbs.db
.DS_STORE

I noticed those too, as ones I'd happily lose without a second glance, while most of the other unmanifested files I'd tread more carefully around.

Would it be reasonable to stash the filenames in a tag-style custom column?
Then we could easily decide how to deal with various files as we discover them ...

[kiwidude]

Originally Posted by capnm

I noticed those too, as ones I'd happily lose without a second glance, while most of the other unmanifested files I'd tread more carefully around.

Ok, I think I see where you and drMerry are going with this. There are files that you would "quite happily lose regardless of whether they are manifested or not", and "the rest".

Calibre bookmarks I think has to remain separately treated as there are users who do read their books in the calibre viewer and want to retain their last read position etc. So while many users would quite happily lose them, there are a number who would not.

iTunesArtwork remains a question mark. Like I said in a query I ran across around 18000 ePubs I didn't have a single one where the iTunesArtwork was manifested. So my natural instinct is to nuke them but as drMerry has found one situation I remain open to hear anything more concrete on that.

That leaves the iTunes plist files, .DS_Store and thumbs.db files which are all just "trash". As drMerry points out they may have been included in your manifest by a user running a tool incorrectly on them or through ignorance.

So perhaps the "Remove iTunes plist files" option is renamed to something like "Remove unneeded junk files" or some better name. So that way a user can always leave it turned on if they so choose and not have to tick three checkboxes. Potentially iTunesArtwork goes into that category one day too, and Calibre bookmarks remains separate.

Would it be reasonable to stash the filenames in a tag-style custom column?
Then we could easily decide how to deal with various files as we discover them ...

This was why I added the log capability to the latest version of Quality Check. Copy the log results to Notepad++ or wherever, then run Modify ePub appropriately on the books shown ticking only what you feel safe to on each book.