[Discussion] Supplementary files

[BetterRed]

I went off topic in this thread ==>> [Enhancement] calibre:// show-details option, on the subject of 'attachments'.

I'll copy the relevant posts from the other thread to here in a moment. An issue I had Count Pages has triggered some further thoughts.

If Book folders had an optional "Attachments" sub folder, perhaps RTF and image files could be stored in it and referenced relatively, might need a book_attachments:// protocol. It would also solve the perennial problem of where to put supplementary material.

Added: maybe the Link field in Authors and other category columns should become a Links field - i.e. a list of <a href="xxx://whatever">label<\a> links

BR

[chaley]

Originally Posted by BetterRed

If Book folders had an optional "Attachments" sub folder, perhaps RTF and image files could be stored in it and referenced relatively, might need a book_attachments:// protocol. It would also solve the perennial problem of where to put supplementary material.

BR

This isn't related to meta-metadata, which has no 1-1 relationship with books. It would require some alternate folder structure and all that entails, such as dealing with deletes and renames.

[ownedbycats]

I believe SQLite supports blob (or none?) data but that probably isn't too practical

[BetterRed]

Originally Posted by chaley

This isn't related to meta-metadata, which has no 1-1 relationship with books. It would require some alternate folder structure and all that entails, such as dealing with deletes and renames.

Putting the meta-metadata issue aside, IMO all it would require of calibre is a tool to create and open a book's Attachments folder and recognise it's existence - i.e. not flag it as an error in Check library. At the moment kludges like adding playlists, or spoofing archive formats are used.

Have to dash

BR

[ownedbycats]

Originally Posted by BetterRed

Putting the meta-metadata issue aside, IMO all it would require of calibre is a tool to create and open a book's Attachments folder and recognise it's existence - i.e. not flag it as an error in Check library. At the moment kludges like adding playlists, or spoofing archive formats are used.

Have to dash

BR

What about handling of the filenames? Should it keep them intact, so that you could have multiples of the same of the same extension?

[BetterRed]

Originally Posted by BetterRed

Putting the meta-metadata issue aside, IMO all it would require of calibre is a tool to create and open a book's Attachments folder and recognise it's existence - i.e. not flag it as an error in Check library. At the moment kludges like adding playlists, or spoofing archive formats are used.

Have to dash

BR

Over the past decade I've seen many more threads asking how other users store supplementary files than I've seen threads asking how other users store data about authors, series, publisher's etc.

Originally Posted by ownedbycats

What about handling of the filenames? Should it keep them intact, so that you could have multiples of the same of the same extension?

In my view the filenames and types would be defined by the user - e.g. 6 epubs, 5 jpgs, 15 txt files, and an mp4 video would be fine. But perhaps no subfolders (real or symbolic).

The actions I envisage calibre performing on the Attachments folder are….

• New Functions
• Create - fixed name (Attachments, Pièces jointe, etc), set a flag in book table row.
• Open - in platform file manager (File Explorer, Finder, Nautilus etc).
• Save - to "<Library>/{author}/{title}/Attachments" in user selected location.
• Delete (hard) - unset a flag in book table row.
• Existing functions (perhaps after ensuring Attachments contained no subfolders…) essentially what's done with the other book folder contents (cover.jpg, metadata.opf and format files) as a collective
• Move it when an author or title is changed.
• Remove it when a book is removed**
• Copy it, and optionally delete it, when the Copy to Library feature is used.

[BetterRed]

So, what happened in Count Pages.

I created a 'book' by adding a CBZ file to a calibre library in which there were almost 200 books with a lone CBZ. But for this 'book' I had a number of small MP4 video clips (animations) I wanted to store with alongside the CBZ so I put them in a RAR and added that to the book.

I then ran the Count Pages (CP) plugin - it didn't update my Pages/#pages custom column It had worked on every other book in that library - difference was none of them had a CBZ and a RAR. So I opened the book folder and renamed the RAR to BAR, and ran CP again - the page count column was updated. I then renamed the BAR back to RAR.

I reported the issue in the CP thread.

When a new version of CP was released, the problem was still evident; assuming it had been overlooked (there had been a flurry of other posts immediately after mine) I posted a reminder.

The response was "Your problem is caused by you using .RAR files - quite simply, don't ".

So I renamed it back to BAR, set Windows to open BAR files with WinRAR, and added a bar.png (WinRAR's icon) to %AppData%\calibre\resources\images\mimetypes.

Which made me think - why not have a calibre tool that can create a zip-like container for a book with a fixed name, e.g. book.attachments (a'la cover.jpg & metadata.opf) with options for basic file operations (add, delete, rename, OS open with), and maybe move files from the book folder into the book.attachments archive and vice-versa. And unpack to a folder and pack a folder as an book.attachments container.

Could it be a plugin. Have to dash - again. pacemaker grease and oil change.

BR

[kovidgoyal]

This is on my long term TODO list. It will likely be a folder that calibre will move around/delete/export in save to disk. Beyond that you will use the OS file manager to manage it. Then main blocker is as usual windows and its absurd mandatory file locking and path length limitations which make implementing moves in a performant and robust manner (avoiding file copying) particularly hairy.

In particular these wont have database entries, which means you wont be able to search for books having them or particular files or similar.

[BetterRed]

Windows file locking is not mandatory, it's up to the application to use or not use.

Example: if I change the Author of a book whilst Notepad++ has the book's TXT format file open, it pops an error message right after calibre does the move, viz:



Sigil & Calibre Editor are similar: they pop an error message when the EPUB that is being edited is saved with a Ctrl+S.

I know I suggested an "Attachments" folder, but on thinking about it I wondered if it could open a can of worms if users were allowed to put .lnk files, windows reparse points (symlinks, folder junctions, mount points etc), hardlinks etc, into the folder.

BR

[ownedbycats]

Shortcut files did use to add directly (I remember running into this when adding a manual from the start menu), but this was changed.

[capink]

For books that have a supplementary material, I use the Action Chains plugin to handle and open the folder/file as follows:

• Tag the books with a tag e.g. supplement
• I add the supplementary material inside a folder named after the book id. This folder resides in a parent folder called calibre_supplements
  
  

  /home/user_name/calibre_supplements/book_id

• In Action Chains plugin, I create a chain the contains an Open With action. In the action settings, I choose my favorite file manager in the binary box, and choose the template option for the file path with the following template:
  

  program:
      parent_dir = '/home/user_name/calibre_supplements';
      if 'supplement' inlist $tags then
          strcat(parent_dir, '/', $id)
      fi

  I also set the following condition so that the chain runs only if one book is selected, and the book must be tagged as "supplement"
  
  

  program:
      if selection_count() == 1 && 'supplement' inlist $tags then
          'true'
      fi

  
  in the condition's datatype choose "text", and in the value box the value must be "true"

Notes:

• To know the book id go to preferences > book details > displayed metadata > tick the box for "id". Now you can see the book id in the book details.
  
  If you don't want to bother with that, you can create a chain to automatically create the directory with the book id for you. Simply create a chain with a Chain Variables action that contains the following template:
  

  python:
  def evaluate(book, context):
      import os
      parent_dir = '/home/user_name/calibre_supplements'
      target_dir = os.path.join(parent_dir, str(book.id))
      try:
          os.makedirs(target_dir)
      except FileExistsError:
          pass
      return ''

• You can combine the two chains above into one chain that creates the supplement folder (if it is not already created), then open it.
  
• You can get creative with the template to make it look inside the folder and open specific filetype(s) with your program of choice.
• Action Chains allows you to bind the chain to keyboard shortcut, so you can easily open the folder/file with one click.

Edit: I have never gone through a restore process. My presupposition here is that it restores the ids of the books as well. (seeing that the book folders have the book is in the folder name)

So, before others adopt a system like that, it is worth knowing what happens with books ids and how to make this compatible with calibre restore

[BetterRed]

Originally Posted by ownedbycats

Shortcut files did use to add directly (I remember running into this when adding a manual from the start menu), but this was changed.

You can add a Windows shortcut (.lnk file) to an existing book by copying it to the clipboard in Windows File Explorer (Ctrl+C) and pasting into the book via the Add books->Add files to selected book records from clipboard option - which for me is Alt+Shift+V.

If you 'View' the LNK file, Windows will open the target in the default program for the link target file type - e.g. a shortcut to a PDF in Acrobat, to an ODT in Writer, to a folder in File Explorer.

One of the downsides is that there's no visibility of the target file type within calibre - you need to open the book folder to see that. But if you only used shortcuts for supplementary folders you could find a png of a folder in chains and put that in the mimetype folder as lnk.png

BR

[BetterRed]

Originally Posted by capink

<snip>

Edit: I have never gone through a restore process. My presupposition here is that it restores the ids of the books as well. (seeing that the book folders have the book is in the folder name)

So, before others adopt a system like that, it is worth knowing what happens with books ids and how to make this compatible with calibre restore

AFAIK - a copy to library will give a book a new id, but they survive a change of Title or Author (because it doesn't require creating a new row in a metadata.db Books table). Not sure about 'restore', it's a manual process and it might depend on where from and how.

BR

[kovidgoyal]

Originally Posted by BetterRed

Windows file locking is not mandatory, it's up to the application to use or not use.

No, it is mandatory. On windows, all files are locked as soon as they are opened, unless the application specially opts to use a file sharing mode. And both applications have to use the file sharing mode when opening files for sharing to work.

In contrast in POSIX file locking is advisory, opening files does not lock them unless the application asks for them to be locked and both applications have to ask for the lock for locking to work.

[BetterRed]

Quick update to where I'm at with this - I am bundling the files I want to attach to a 'book' into a .RAR file with WinRAR, renaming it to .CAL-ATTACH and adding that to the book. I have configured Windows to open CAL-ATTACH with WinRAR.

I can add, rename and remove files, I can open files with the default programs i.e. epub's with ebook-viewer, pdfs with PDFXchange, DOCX with Word (and replace on Save), images with IrfanView, video's with VLC etc

The thing I miss most is Open With… which means I can't open an epub with Sigil or a cbz with WinRAR - I can drag them out of the CAL-ATTACH, do what I want and drag them back into the CAL-ATTACH. If anyone knows of Windows archiver with an Open With... feature perhaps they could let me know.

Some might find this 10+ year old thread of interest ==>> Supplementary material.

BR