Help with non-standard characters in directory names

[Terry Brown]

I have been working on a python program that retrieves information from my Calibre database and acts as a bookmarking tool for syncing my reading location across multiple devices I use to read multiple formats of ebooks, but that don't sync very reliably.

As part of this, I have accessed cover files within the Calibre library file structure.

I am using macOS. In python, I am using the pillow library (from PIL import Image) to read the cover image, convert to a png thumbnail for my use.

The problem is that characters like single and double quotes, commas, non-ascii characters, and so on mean that I cannot get to some of the directories Calibre creates. A good example is the recent book "Ring Shout" by P. Djèlí Clark--the non-standard characters in the author name mean that the PIL library calls using the path I create don't work.

I have tediously fixed all the current problems by manipulating the author names and book titles in Calibre to remove or change the offending characters--not a good solution.

Anyone know how can I manage this in the Python code?

[DNSB]

Originally Posted by Terry Brown

I have been working on a python program that retrieves information from my Calibre database and acts as a bookmarking tool for syncing my reading location across multiple devices I use to read multiple formats of ebooks, but that don't sync very reliably.

As part of this, I have accessed cover files within the Calibre library file structure.

I am using macOS. In python, I am using the pillow library (from PIL import Image) to read the cover image, convert to a png thumbnail for my use.

The problem is that characters like single and double quotes, commas, non-ascii characters, and so on mean that I cannot get to some of the directories Calibre creates. A good example is the recent book "Ring Shout" by P. Djèlí Clark--the non-standard characters in the author name mean that the PIL library calls using the path I create don't work.

I have tediously fixed all the current problems by manipulating the author names and book titles in Calibre to remove or change the offending characters--not a good solution.

Anyone know how can I manage this in the Python code?

My suspicion is that calibre is already sanitizing (Latinizing??) the non-standard characters to create the filenames. You may have to duplicate this process when creating the file path.

[kovidgoyal]

If you want to manipulate the calibre library from a script use calibredb https://manual.calibre-ebook.com/gen...calibredb.html

If you want to use python you can run your python script with

calibre-debug

and then use calibre's python apis instead of calibredb. See the manual for documentation of the database apis.

[Terry Brown]

Thanks for the replies.

I do use calibredb, called from python. I use 'list -f all --for-machine' as options to get the entire calibre database as json. Then I create the path to the cover file for a book by concatenating the library path on my Mac with the 'authors', 'title', 'id' fields taken directly from the json data.

But when I do so, the path created has characters in it that result in a "file not found" error when I use the path to call Image.open():

im = Image.open(calibrecoverpath)

That's why I'm confused. The files on disk are accessible to me from Finder or the command line, so the names must be sanitized, but the data from calibredb that I get as json is apparently not.

I manipulate (both read and write) the calibre database using calibredb from within my program in many other ways. In this case, I just want to grab the cover file so I can display it with my program when I show reading stats on a book. I don't see a way to directly get the path to the cover file from calibredb, so I use the individual fields (author, title) to create the path.

What should I do instead?

[Terry Brown]

By the way, I will look into calibre-debug . . .

[jhowell]

Originally Posted by Terry Brown

...Then I create the path to the cover file for a book by concatenating the library path on my Mac with the 'authors', 'title', 'id' fields taken directly from the json data.

As others have mentioned the path for the book may not exactly match the author and title fields because they may contain disallowed characters or need to be truncated.

The "cover" field has the correct full path to the cover file.

[qwertyuio]

As has been pointed out, calibre sanitizes the title/author when creating directories as not all filesystems support all characters. Since the json returned by calibredb doesn't not provide the actual path, the method I use in my python scripts is walk the entire library and identify the books by id. See the attached python function I use.

[kovidgoyal]

And the formats field has the paths to individual book files.

[qwertyuio]

$ calibredb list -f formats,cover 
id  formats cover 
1   [PDF]  1
2   [PDF]  1
3   [EPUB] 1

$ calibredb list -f formats,cover --for-machine
[
  {
    "cover": 1,
    "formats": [
      "PDF"
    ],
  },

Is there an option I'm missing?

[kovidgoyal]

Presumably you are on an old calibre version.

calibredb list -f formats,cover --for-machine -s id:2017                                                                                *
[
  {
    "cover": "/home/kovid/test library/calibre/The Hindu (2017)/cover.jpg",
    "formats": [
      "/home/kovid/test library/calibre/The Hindu (2017)/The Hindu - calibre.mobi"
    ],
    "id": 2017
  }
]%

And note that this wont work when connecting via the calibre content server.

[qwertyuio]

Yes, I use the content server because I want to be able to use calibre while a script is running in the background.

[kovidgoyal]

The content server does not respond with paths, as they are both meningless over a network and an information leak about the layout of files on the server.

[Terry Brown]

Thanks, all, for the help with this. I, too, was using the content server, so the 'cover' field did not contain the path. By turning off the content server I can get exactly what I want.