Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 05-28-2011, 08:01 PM   #31
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
Quote:
Originally Posted by kiwidude View Post
??? What do you mean? As I understand it this file is only used for displaying a cover in iTunes. Either you are interested in putting your ePubs in iTunes (in which case you won't bother removing any iTunes files), or you aren't (in which case removing any seems perfectly safe and sensible). There is no code in Calibre I could find that looks for this filename.
The file is an image file. So if the cover in the epub is set to use this file, your cover wil be broken (not a real problem if it is allready set in calibre, but you will loose your option to set the cover from the file.

So to use this save, you should be sure the file is not manifested, but also not directly linked from one of the (x)html files
drMerry is offline   Reply With Quote
Old 05-29-2011, 09:50 AM   #32
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Can anyone else verify this? As every single ePub I have that has an iTunesArtwork file does not have that file listed in their manifest. It has just been inserted by iTunes in the same way that it inserts the .plist file.

I understand that "in theory" someone could create an ePub and modify their manifest to point the cover to it, but I am more interested in this "does this really happen" than the theoretical. In theory people can do all sorts of dumb-ass things to their ePubs.
kiwidude is offline   Reply With Quote
Advert
Old 05-30-2011, 09:53 AM   #33
capnm
Groupie
capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'
 
Posts: 156
Karma: 10001
Join Date: Feb 2011
Device: sony
re itunes art

Quote:
Originally Posted by kiwidude View Post
what I am going to do with those in both this and the QualityCheck plugin is just treat them as "iTunes files" rather than have two separate functions for plists and artwork. Had I know about them previously it would have been the initial behaviour.
fwiw I completely agree with this approach.
Sorry I hadn't noticed the artwork when I first brought this up.
capnm is offline   Reply With Quote
Old 05-30-2011, 09:54 AM   #34
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
v0.2 Beta

Changes in this release:
  • Add option to add/update calibre jackets
  • Add option to remove non dc: metadata elements
  • Add option to remove iTunesArtwork files
  • Rename Select none to Clear all on dialog

The adding/updating of jackets was an interesting one. At the moment I have made a couple of assumptions. Primarily that the "output profile" you choose is the same as that set in your Calibre default (unlike when you do a conversion you have the option to override this). If further down the track this is seen as something you may need to also override in this plugin then I can add a dropdown to the modify dialog.

Similarly you do not have the option to manually "override" stuff like title and authors like you could do with a conversion. So it uses the metadata you have in your library which is the same that a conversion would default to.

The removal of non dc namespace elements is in response to the request above. This removes some of the obvious traces of a Calibre conversion or editing in Sigil etc. I can't think of a reason to use this other than if you were publishing your ePub but as it sounds useful for that alone I have included it.

As I mentioned above I still need to be convinced on the iTunesArtwork files needing to be treated differently to the plist files, as I don't have an example ePub which includes the artwork in the manifest. For now at least it is a separate checkbox you can choose in this plugin. If there is some publishing standard/application that does create such manifest entries for iTunesArtwork files and someone can PM me a link to an ePub example or post the .opf manifest then I will enhance this to ensure any such entries are also removed.
Attached Thumbnails
Click image for larger version

Name:	Screenshot_1_Options.png
Views:	426
Size:	35.3 KB
ID:	72082  

Last edited by kiwidude; 06-26-2012 at 01:56 PM.
kiwidude is offline   Reply With Quote
Old 05-30-2011, 10:42 AM   #35
capnm
Groupie
capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'capnm knows the difference between 'who' and 'whom'
 
Posts: 156
Karma: 10001
Join Date: Feb 2011
Device: sony
Could you preserve the file dates (timestamps) when you unzip/rezip the epub?

Two reasons:
One is completely OCD, but if all I'm doing is removing unwanted files somebody else (iTunes/calibre) added to my retail epub, I'd rather keep the html original, including timestamp.

The other is it makes for an interesting anomaly -- if I make a copy of an epub, (which then matches in your binary compare), strip the bookmarks file out of both copies (at slightly different times), the copies no longer compare:
even though all the content (and hashes) are identical the epub itself is different (and hashes different) due to the timestamps differing.


-- This brings up another thought -- would it be practical to add a "fuzzy" option to your binary compare, something like you open the epub and check the hash of the largest folder for a match?
(and should I bring this question up in the Duplicate Check thread?)

Last edited by capnm; 05-30-2011 at 10:49 AM. Reason: afterthought
capnm is offline   Reply With Quote
Advert
Old 05-30-2011, 11:04 AM   #36
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by capnm View Post
Could you preserve the file dates (timestamps) when you unzip/rezip the epub?
At the moment I am using the same approach that Kovid has in the Tweak ePub and epub-fix features. This calls the calibre.utils.libunzip.extract() function. It appears that this function does not retain the timestamps from the zip for the extracted files so all suffer from the same problem.

If Kovid can fix this or suggest an alternative way to extract the file that preserves the timestamps on the files then I should hopefully be able to do something. I agree it would be desirable if possible so that you can see only the files that were truly touched.

Quote:
-- This brings up another thought -- would it be practical to add a "fuzzy" option to your binary compare, something like you open the epub and check the hash of the largest folder for a match?
(and should I bring this question up in the Duplicate Check thread?)
You are correct that it is a question for the Find Duplicates thread, but I will give you an answer here anyways. I have no interest at this point in changing the plugin to start looking at ePub content - it will dramatically slow it down by many orders of magnitude and opens the door for a number of other issues which I am quite happy to avoid.
kiwidude is offline   Reply With Quote
Old 05-30-2011, 11:06 AM   #37
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,171
Karma: 16228536
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
Hi kiwidude,

I have a suggestion for your consideration re: the XPGT margins option.

The scenario is for a retail epub (minus DRM) which has a .xpgt file and has no need for a calibre epub-epub conversion. The epub will either have:
- margins in the .xpgt plus margins in the epub .css
- margins only in the .xpgt

For the former I can see that you would want to remove the .xpgt margins completely.

For the latter, if you remove the .xpgt margins, you would end up with no margins at all, which can be rather cramped on the device. In this case it might be nice to have the option to set .xpgt margins to match those specified in Prefs - Common Options - PageSetup. This should result in the same page margins you would have got if you'd done an epub-epub conversion, but without the conversion 'risk'.
jackie_w is offline   Reply With Quote
Old 05-30-2011, 11:08 AM   #38
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The python Zip library does not preserve timestamps when extracting from zip files. While I could patch it, I dont care enough to do so. If you do, the code is in calibre.utils.zipfile.

EDIT: If you do decide to add that functionality, remember that zip files dont store time zone into on their timestamps. So you essentially have no way of knowing what time the timestamp in a zip file actually is

Last edited by kovidgoyal; 05-30-2011 at 11:13 AM.
kovidgoyal is offline   Reply With Quote
Old 05-30-2011, 11:27 AM   #39
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by jackie_w View Post
Hi kiwidude,

I have a suggestion for your consideration re: the XPGT margins option.

The scenario is for a retail epub (minus DRM) which has a .xpgt file and has no need for a calibre epub-epub conversion. The epub will either have:
- margins in the .xpgt plus margins in the epub .css
- margins only in the .xpgt

For the former I can see that you would want to remove the .xpgt margins completely.

For the latter, if you remove the .xpgt margins, you would end up with no margins at all, which can be rather cramped on the device. In this case it might be nice to have the option to set .xpgt margins to match those specified in Prefs - Common Options - PageSetup. This should result in the same page margins you would have got if you'd done an epub-epub conversion, but without the conversion 'risk'.
Hi Jackie,

I guess the question I have is how do you "know" that there are only .xpgt margins available? Is this something that only manual visual inspection of the css can reveal? In which case presumably a separate checkbox option would suffice for this. I guess my concern would be that if it is something you can only determine by using Tweak ePub, is it not just as easy to manually edit the .xpgt file at the same time?
kiwidude is offline   Reply With Quote
Old 05-30-2011, 11:31 AM   #40
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by kovidgoyal View Post
The python Zip library does not preserve timestamps when extracting from zip files. While I could patch it, I dont care enough to do so. If you do, the code is in calibre.utils.zipfile.

EDIT: If you do decide to add that functionality, remember that zip files dont store time zone into on their timestamps. So you essentially have no way of knowing what time the timestamp in a zip file actually is
Thx for confirming the above Kovid. Since other zip programs manage to work around the timezone issue presumably there must be some sort of compromise?
kiwidude is offline   Reply With Quote
Old 05-30-2011, 11:32 AM   #41
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,776
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by kiwidude View Post
Thx for confirming the above Kovid. Since other zip programs manage to work around the timezone issue presumably there must be some sort of compromise?
The innocence of youth. They just assume the time is in whatever the local timezone of the machine running the extractor is.
kovidgoyal is offline   Reply With Quote
Old 05-30-2011, 11:49 AM   #42
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,171
Karma: 16228536
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
Quote:
Originally Posted by kiwidude View Post
Hi Jackie,

I guess the question I have is how do you "know" that there are only .xpgt margins available? Is this something that only manual visual inspection of the css can reveal? In which case presumably a separate checkbox option would suffice for this. I guess my concern would be that if it is something you can only determine by using Tweak ePub, is it not just as easy to manually edit the .xpgt file at the same time?
I was not really thinking of automated decision-making by the plugin, just a user decision on an epub by epub basis. The first a user may know of the problem is when they try and read it on the device. The plugin would give them an option to correct and resend. I was thinking of it being targetted at less technical 'non-tweaking' users who aren't comfortable with the surgical approach.
jackie_w is offline   Reply With Quote
Old 05-30-2011, 12:02 PM   #43
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by kovidgoyal View Post
The innocence of youth. They just assume the time is in whatever the local timezone of the machine running the extractor is.
Haha. Well I guess I don't know what the downside of doing that is. Since we are not actually "using" the date/times extracted, we just want them to have the same value when put back into the zip file as they have when extracted.

Bearing in mind my complete ignorance on such dates and times, I tried adding these two lines to zipfile.py at line 1126:
Code:
                mtime = time.mktime(member.date_time + (0,0,0))
                os.utime(targetpath, (mtime, mtime))
This gets me "close", but every time I zip the file back up the hour increments by one. Or is that just an example of the timezone issue?
kiwidude is offline   Reply With Quote
Old 05-30-2011, 12:21 PM   #44
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by jackie_w View Post
I was not really thinking of automated decision-making by the plugin, just a user decision on an epub by epub basis. The first a user may know of the problem is when they try and read it on the device. The plugin would give them an option to correct and resend. I was thinking of it being targetted at less technical 'non-tweaking' users who aren't comfortable with the surgical approach.
I'm pretty ignorant about the xpgt files and their purpose so forgive my dumb questions on this, but I want to make sure I have this right.

Does an xpgt file override css? So if you see an ePub squashed against the sides and you want to adjust it, you are saying that modifying the xpgt file will always "fix" the issue, regardless of what css is or is not present?

What if the ePub has no xpgt file? Would this function then be expected to generate and add an xpgt file to the ePub?
kiwidude is offline   Reply With Quote
Old 05-30-2011, 01:07 PM   #45
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,171
Karma: 16228536
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
Quote:
Originally Posted by kiwidude View Post
I'm pretty ignorant about the xpgt files and their purpose so forgive my dumb questions on this, but I want to make sure I have this right.
As you're a Kindle owner I'm more than impressed that you've been prepared to take on this epub project at all

Quote:
Originally Posted by kiwidude View Post
Does an xpgt file override css? So if you see an ePub squashed against the sides and you want to adjust it, you are saying that modifying the xpgt file will always "fix" the issue, regardless of what css is or is not present?
I stand to be corrected on this, but to the best of my knowledge the xpgt and css margins are cumulative. The css margins are visible in the calibre Viewer, the xpgt ones are not (this would be a nice Viewer enhancement but not relevant to this discussion).

I believe the xpgt margins are page-oriented. The xpgt is referenced in each and every epub html file in a <link> statement in the heading. To the best of my knowledge, if the epub css has left/right margins set to zero or no left/right margins at all, then non-zero margins in the xpgt will still result in whitespace around the text on every page when viewed on the device.

Calibre conversion PageSetup margins seem to be applied as follows:
- left/right are placed once, in the epub's css file in the css selector class used on the <body> tag
- top/bottom are placed in the heading of each and every epub html file between <style> tags as a
Code:
@page{margin-bottom: mpt; margin-top: npt}
Quote:
Originally Posted by kiwidude View Post
What if the ePub has no xpgt file? Would this function then be expected to generate and add an xpgt file to the ePub?
Probably not. It sounds like a lot of work and on the whole people find xpgt files a PITA and may not thank you for creating extra ones.
jackie_w is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Any web-to-epub plugin for internet browser? bthoven ePub 7 07-10-2011 05:14 AM
[Old Thread] Reading epub on viewer inexplicably changes the time stamp of epub greenapple Library Management 20 03-19-2011 10:18 PM
Easy way to modify thread subscription emails in bulk? snipenekkid Feedback 11 02-06-2011 03:47 AM
Another plugin dev question DiapDealer Plugins 2 12-11-2010 01:46 PM
Epub plugin dev DiapDealer Plugins 15 11-12-2010 09:36 AM


All times are GMT -4. The time now is 06:49 PM.


MobileRead.com is a privately owned, operated and funded community.