Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 05-25-2018, 02:05 PM   #1
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,216
Karma: 16534894
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
container.mi and unicode

I'm in the process of creating a User Interface plugin as a calibre version of Doitsu's new Sigil plugin (convert selected epub text files to MP3 using the MS Windows Speech API and the LAME encoder).

I'm still doing initial testing so am still running my .py scripts via calibre-debug in a Windows .bat file rather than via plugin. Consequently, I don't have access to the calibre library metadata ATMO so am trying to extract the few items I want from the container.mi object.

My problem is in understanding whether a container.mi field contains unicode or something else. For example this print statement
Code:
print('authors:', container.mi.authors, '\ntitle:', container.mi.title)
results in this in my CMD box
Code:
authors: [u'Yrsa Sigur\xf0ard\xf3ttir']
title: A ‘unicode’ title Yrsa Sigurðardóttir
'title' looks like a unicode string, but 'authors' looks like a list of non-unicode items.

Please can you advise how I should be accessing the container.mi data to make sure I always end up with unicode?

I could do what the Sigil plugin does and extract the data directly from container.opf but I'm sure container.mi has already done a far more robust job of that than anything I could come up with.

The metadata will be used to populate the MP3 tags.

I can attach a small test epub if necessary.

Last edited by jackie_w; 05-25-2018 at 02:12 PM. Reason: better example
jackie_w is offline   Reply With Quote
Old 05-25-2018, 02:56 PM   #2
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,216
Karma: 16534894
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
After further investigation the contents of container.mi may not be my problem. The problem may arise when supplying the args to LAME.exe/subprocess.

I definitely have a problem that Sigil is correctly populating unicode MP3 tags, but when I supply what looks like the same args to LAME.exe/subprocess via calibre I get messed up unicode MP3 tags.

I'll come back when I have a better question to ask.
jackie_w is offline   Reply With Quote
Advert
Old 05-25-2018, 10:24 PM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,397
Karma: 23798586
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
subprocess in python 2 on windows cannot handle unicode arguments if you are using a non-utf-8 version of windows. You will need to use CreateProcess see gui2/open_with.py for an example fo doing that.
kovidgoyal is online now   Reply With Quote
Old 05-26-2018, 06:03 PM   #4
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,216
Karma: 16534894
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
Thanks for pointing me in the right direction.

This low-level OS stuff is way out of my comfort zone. It's taken me forever and a lot of googling to work through it but I think I finally have something which can run lame.exe, handles unicode MP3 tags and produces a returncode using
Code:
retcode = win32event.WaitForSingleObject(process_handle, win32event.INFINITE)
I have a few questions:
  1. Your example in gui2.open_with has the dwCreationFlags parameter set as win32con.CREATE_DEFAULT_ERROR_MODE | win32con.CREATE_NEW_PROCESS_GROUP | win32con.DETACHED_PROCESS.

    I think I need DETACHED_PROCESS for my purposes but please could you explain the other two.
  2. I've just copied your use of 'with sanitize_env_vars()' verbatim without understanding it. Is this OK?
  3. I didn't even know there were utf-8/non-utf-8 versions of Windows. This new CreateProcess method is working for my Win10 x64 setup but will one-size fit all Windows users.
  4. I don't think I need to return stdout/stderr for this particular plugin but I may in the future. Please could you give me a a few pointers how to proceed.
jackie_w is offline   Reply With Quote
Old 05-26-2018, 11:10 PM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,397
Karma: 23798586
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
1. They are documented here: https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx You should use DEFAULT_ERROR_MODE. You dont need DETACHED_PROCESS

2. Yes that's fine

3. Yes the point of using CreateProcessW is that it uses the unicode windows API which passes all data between the system and the application in utf-16 encoding, regardless of the codepage that windows itself uses

4. Lets cross that bridge when you come to it. Basically the easiest way would be to simply redirect the stdout/stderr of the process to a file. Then after the process finishes read the contents of the file.
kovidgoyal is online now   Reply With Quote
Advert
Old 05-27-2018, 10:19 AM   #6
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,216
Karma: 16534894
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
Quote:
Originally Posted by kovidgoyal View Post
3. Yes the point of using CreateProcessW is that it uses the unicode windows API which passes all data between the system and the application in utf-16 encoding, regardless of the codepage that windows itself uses
Just when I thought I had this working ... Are you saying I should be using ???.CreateProcessW rather than win32process.CreateProcess as used in open_with.py?

Quote:
Originally Posted by kovidgoyal View Post
1. They are documented here: https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx You should use DEFAULT_ERROR_MODE. You dont need DETACHED_PROCESS
Thanks for that.
jackie_w is offline   Reply With Quote
Old 05-27-2018, 10:33 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,397
Karma: 23798586
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No, IIRC win32process uses the correct Createprocess call internally when passed unicode arguments.
kovidgoyal is online now   Reply With Quote
Old 05-27-2018, 10:49 AM   #8
jackie_w
Grand Sorcerer
jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.jackie_w ought to be getting tired of karma fortunes by now.
 
Posts: 6,216
Karma: 16534894
Join Date: Sep 2009
Location: UK
Device: Kobo: KA1, ClaraHD, Forma, Libra2, Clara2E. PocketBook: TouchHD3
OK, I'm glad about that. Thanks for your help
jackie_w is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
missing container.xml error George H Editor 5 11-26-2022 09:43 AM
Recreating Container.xhtml SigilBear Sigil 42 06-27-2017 05:32 PM
container.xml problems SigilBear Sigil 4 06-11-2017 09:43 PM
Container methods, various scenarios jackie_w Development 15 11-09-2015 02:32 PM
META-INF/container error afpeter ePub 9 07-23-2013 01:04 AM


All times are GMT -4. The time now is 06:11 AM.


MobileRead.com is a privately owned, operated and funded community.