Adding a new ebook format

[shentar]

Is there a way to make calibre to recognise a new extension as an ebook (eg recognize .nef as New Ebook Format)?

I know I can add files of any format by explicitly selecting them in the "Add books" dialog but if you have a lot of them in a directory tree calibre will not find them as it does not know that .nef is now an ebook. I am hoping there is a formats.xml or similar file somewhere which controls this behaviour but if there is I can't find it.

Thanks,

[JSWolf]

.nef is not an eBook format. NEF is Nikon Electronic Format fir RAW image files. So nobody is going to be writing code for Calibre to support Nikon RAW images.

[jgoguen]

Assuming you used .nef as an example you can check out the KePub plugins code at https://github.com/jgoguen/calibre-kobo-driver. I can't remember where exactly, but one of the files there defines the '.kepub' extension so calibre can recognize KePub books made through the conversion plugin.

[BetterRed]

Moderator Notice
@shentar - read the "PLEASE READ..." sticky at top of Development sub-forum.

can you indicate in your next post whether you're prepared to any necessary development for this.

AFAIK - there is no way to add support for new a format without changing the calibre code and/or developing a driver as per jgoguen's post. Convertible formats require Input and Output plugins.

You can add non ebook files via the calibredb command using the 'add' option. So you could write a script/batch file to walk your directory tree and add the .nef files via that command.

Originally Posted by JSWolf

.nef is not an eBook format. NEF is Nikon Electronic Format fir RAW image files. So nobody is going to be writing code for Calibre to support Nikon RAW images.

- I have a bunch of them. But I don't think there's anything to stop someone else using it for another purpose Ψ²

BTW is this yours File extension WOLF

BR

[eschwartz]

Originally Posted by BetterRed

BTW is this yours File extension WOLF

BR

Well, I certainly won't write code to support him.


@shentar -- use the Adding books dialog, sorted by filetype, and shift-click to select them all. You will need to set the filter to All files (*) in order to see them.

[BetterRed]

Originally Posted by eschwartz

@shentar -- use the Adding books dialog, sorted by filetype, and shift-click to select them all. You will need to set the filter to All files (*) in order to see them.

@eschwartz - the .nefs are in a tree. So to do bulk select you would need a file manager that provides an Add from single directory compliant flat view - wouldn't you?

AFAIK Windows Explorer and Finder don't have this feature.

BR

[eschwartz]

Ah -- didn't notice that. Time to break out the bulk file renaming utilities.

Or to be honest a script to walk the tree and use `calibredb add` would be just annoying and more to the point.

[shentar]

Thanks guys.

Development work : I affirm that I am willing to do any/all development work for this project, especially given the nature of what I want to do.

I am investigating the possibility of misusing (repurposing?) calibre as a front end to my fairly extensive video library. I currently use a WD TV Live box which plays files directly from my NAS. The upside of this is that I don't need to have a PC running. The downside is that I can either use the media library functions on the WD box or simply browse a list of titles. The title list is much faster and more convenient ... until I forget the name. I have set up hard links on the NAS to provide some primitive indexing via lead actor, director, genre, series but it is soul destroying to do this manually. Unfortunately WD have given no indication of upgrading the TV Live to support the more recent video formats (10 bit colour, HEVC and the like) nor do they appear to have any successor products. My plan is to use a small linux computer (intel NUC or even a Raspberry Pi) running calibre and vlc, with the actual calibre library on the NAS. This leaves me with two problems - adding the video files to the library and populating a set of custom columns with director, cast, genre and similar info. If I can add custom "book" types (.mkv, .mp4, .avi etc) then I can use the calibre GUI bulk add to get the actual files into the library, after which I can use a python script to parse my DVD Profiler database and the calibre command line to insert the additional information. As you say, I can parse the file source tree myself but why write code if it is already there and tested

I'm willing to write an import plugin if I have to. There will be no file conversion issues for obvious reasons but in fact it would be useful to take control of the file copy operation so that I can keep my existing master library and use links for the calibre library instead of doing a straight copy. The pointer to the KePub plugin may make life a lot easier.

TBH, in many ways it would be easier to start with DVD Profiler as the front end but there is no natural way to play a film from within DVD Profiler (it expects them to be on disks) plus there is no native linux version and I can't be asked to write plugins for windows code running under wine. Besides, the typing, sorting, series identification etc is much better under calibre.

If this works I will make any plugins and scripts available to anyone who wants them, although I suspect they will need tailoring as this is not intended as a fully general utility, just a one-shot conversion.

Just to confirm, .nef was an example, I should have realised that almost all the three letter extensions are already taken.

[BetterRed]

Originally Posted by shentar

Development work : I affirm that I am willing to do any/all development work for this project, especially given the nature of what I want to do.

Originally Posted by shentar

Just to confirm, .nef was an example, I should have realised that almost all the three letter extensions are already taken.

The '.ext' file type is often used as a generic, only old never die Norton Commander fans would have any of those

I tried using symlinks sometime back, but I got into trouble with renames (via changes to Title and Author) and functions like Copy to Library - the symlinks would become copies.

I overcame those issues by using Windows .LNK (shortcut) files, but to sidestep Windows shortcut processing I had to add them via Windows Send to or the command line. Which was a bit of pain.

I then used file:\\\blah blah URL's in a long text custom column that I displayed in the Book Details panel - the advantage was that I could have multiple files of the same format in the one 'book'.

I now put the file inks etc into an Evernote note and link to it from calibre with an evernote:\\\blah blah link. Evernote has 'richer' editing facilities than calibre's comments editor - but the main reason is that I can get at them on the web or on my tablet where I can add dictated audio to notes.

BR

[jgoguen]

You want to take software based around and making assumptions about book files and make it work with videos?

That should be interesting, I look forward to seeing what you come up with!

[kovidgoyal]

Simply do a search for .nef in windows explorer in the tree you are interested in, and then drag and drop the results into calibre.

[shentar]

Originally Posted by BetterRed

The '.ext' file type is often used as a generic, only old never die Norton Commander fans would have any of those

I tried using symlinks sometime back, but I got into trouble with renames (via changes to Title and Author) and functions like Copy to Library - the symlinks would become copies.

I overcame those issues by using Windows .LNK (shortcut) files, but to sidestep Windows shortcut processing I had to add them via Windows Send to or the command line. Which was a bit of pain.

I then used file:\\\blah blah URL's in a long text custom column that I displayed in the Book Details panel - the advantage was that I could have multiple files of the same format in the one 'book'.

I now put the file inks etc into an Evernote note and link to it from calibre with an evernote:\\\blah blah link. Evernote has 'richer' editing facilities than calibre's comments editor - but the main reason is that I can get at them on the web or on my tablet where I can add dictated audio to notes.

BR

Symlinks are probably not a good idea for what I want to do. It might work if calibre were running on the same box as the files but the current plan is to have calibre on a small linux box and the files on a Synology NAS. Fortunately the Synology also runs linux and you can log onto it and use it as a limited function linux PC. This gives me the option to use hard links which are more or less indistinguishable at the linux app level from genuine files.
The Evernote technique is an interesting idea, although I don't need it for this as all access will be from within the home. I don't use Evernote currently but several people have recommended it so I should probably revisit it.

Originally Posted by jgoguen

You want to take software based around and making assumptions about book files and make it work with videos?

That should be interesting, I look forward to seeing what you come up with!

Don't expect too much. Most video container files carry little or no metadata of interest concerning their content other than technical info such as format, resolution and the availability of audio & subtitle files. At least initially I have no intention of interrogating the file itself hence my plan to parse my DVD Profiler database. In fact, one of the reasons for using calibre as a starting point is that it provides a neat way of associating an externally stored metadata with the files and searching on it. I'll turn off the option to attempt to extract metadata from the "book" files and to calibre the video files will simply be black boxes.

Originally Posted by kovidgoyal

Simply do a search for .nef in windows explorer in the tree you are interested in, and then drag and drop the results into calibre.

Sigh. I could say "why didn't I think of that" but TBH it has never occurred to me that calibre supports drag'n'drop as an entry method. With a complex piece of software it is too easy to keep doing things the way you always have and not realise there are alternatives. Thanks!

[davidfor]

Originally Posted by shentar

Don't expect too much. Most video container files carry little or no metadata of interest concerning their content other than technical info such as format, resolution and the availability of audio & subtitle files. At least initially I have no intention of interrogating the file itself hence my plan to parse my DVD Profiler database. In fact, one of the reasons for using calibre as a starting point is that it provides a neat way of associating an externally stored metadata with the files and searching on it. I'll turn off the option to attempt to extract metadata from the "book" files and to calibre the video files will simply be black boxes.

If the purpose is to get the metadata from a database to calibre and connected to a file, you might want to look at the Import List plugin. It can take a CSV file and create or modify book entries in calibre. If you can extract the data from DVD Profiler, the plugin might offer a short cut.

[shentar]

Originally Posted by davidfor

If the purpose is to get the metadata from a database to calibre and connected to a file, you might want to look at the Import List plugin. It can take a CSV file and create or modify book entries in calibre. If you can extract the data from DVD Profiler, the plugin might offer a short cut.

I'll look at that. Unfortunately the only obvious export option from DVD Profiler is to an XML file, and if I am going to have to parse that I might as well convert it directly into calibredb commands.
If there is a DVD Profiler plugin for CSV output, on the other hand ...

[BetterRed]

Originally Posted by shentar

Symlinks are probably not a good idea for what I want to do. It might work if calibre were running on the same box as the files but the current plan is to have calibre on a small linux box and the files on a Synology NAS. Fortunately the Synology also runs linux and you can log onto it and use it as a limited function linux PC. This gives me the option to use hard links which are more or less indistinguishable at the linux app level from genuine files.

I think you're confused about hard and sym/soft links. AFAIK hardlinks wont stretch across networks, they have to be on the same partition see What is a hard link? -- by The Linux Information Project.

Which is why I referred to symlinks - which will stretch across file systems and partitions and networks.

Originally Posted by shentar

The Evernote technique is an interesting idea, although I don't need it for this as all access will be from within the home. I don't use Evernote currently but several people have recommended it so I should probably revisit it.

AFAIK there's no official Linux implementation of Evernote, but there are some unofficial ones - Everpad and Nevernote are two of which I'm aware.

Originally Posted by shentar

Sigh. I could say "why didn't I think of that" but TBH it has never occurred to me that calibre supports drag'n'drop as an entry method.

I should have remembered that too, but I don't do much bulk adding these days and with non-ebook files I'm usually adding to an existing book.

You could probably find a utility or a site that will transform XML into CSV suitable for use with the Import List PI - or you could have a look at the possibility of adding XML as as input source to the PI

BR