Calibre "save to disk", "last_modified" field format

[bodiccea]

Hi,

I would like to have the "last modified" modification date for generated file when i "save to disk" instead of the file "last modified date".

I can see 2 solutions:
- change the "last modified date" in generated file after it has been written
- allow to alter the "last_modified" format in "save to disk" options, so I could have for instance "yyyyMMddhhmm", (it is "MMM, yyyy" today). In that case I could make a simple script to change the file modification date from this format information.

Any idea?

Regards,

bodi.

[BetterRed]

bump

@bodiccea - I'm not sure what you're asking, but I am wondering if you're confusing calibre's last_modified/Modified column and the file system's Modified Date.

The first is the date/time that a book's entry in the metadata database was last modified, it cannot be changed. The second is the date/time the contents of a file were last modified, it can be changed via your file browser (File Explorer, Finder etc).

BR

[bodiccea]

Hi, and thanks for your answer.

I know of course the difference of times between the calibre database one, and the exported file one.

I present my books are on a web site, and I would like people to know what has changed (i.e. file modification date should be calibre modification date).

I would like to be able to include both calibre "modification date" date in the save to disk (in the Preferences/import/export/template).
For instance, if I could add "calibre creation date" in format yyyyMMddhhmm, the exported file could be (on linux file system) file system

$ ls -l --time-style=long-iso
-rw-rw-r-- 1 br br 1119273 2018-05-19 15:56 200402232331 - Gallo,Max - Nice.epub

Here 200402032331 would be the modification date in calibre library, and would become part of the created file name (bold).

With a 2 lines script I could change the exported file modification date and rename the file, so that the file would become:

$ ls -l --time-style=long-iso
-rw-rw-r-- 1 br br 1119273 2004-02-23 23:31 Gallo,Max - Nice.epub


You will notice (blue color), that the file modification date is now what I would like to see, so that people can find easily what is new).

Is it more clear ?

Thanks.

PS. Have a look on:
https://ebooks.raoult.com/eng/?C=M;O=A

These books were added in Calibre at different dates for 5-6 years, and it is impossible for people to know if there are some new ones...

[kovidgoyal]

I dont recall if {last_modified} works in save to disk templates or not, it probably does, but if not you can definitely sue either {timestamp} or {pubdate} for the date and ublished columns respectively.

https://manual.calibre-ebook.com/template_lang.html

[bodiccea]

I found a temporary solution.

I added a custom column {#brtime} of type date and built from other values.
Its template is "{timestamp:format_date(yyyyMMddhhmm)}". The issue is {timestamp} is book change, not metadata. For me changing metadata is changing book too.

My "Saving books to disk" template is (one subdir per book language).
{languages}/{#brtime}-{author_sort} - {series:|(| } {series_index:0>2s|#|) - }{title}

Is looks like for instance:
-rw-rw-r-- 1 br br 390974 Aug 24 20:11 201508150000-du Maurier,Daphné - Rebecca.epub

In calibre, Modified column is 20180224021402 (metadata change).
The date in red is book modification time {timestamp} of Calibre (strangely hhmm are 0000 for all of them).
I made a script which gets this date, renames correctly, and changes its "modification time".

Better than a situation where all books were dated Aug 2018.

The script (bash):
----------------------------------------------------------------------
#! /bin/bash
#
# finds all epubs in first level subdirs, which should have the format:
# 201402150000-Arnaud,G.J. - (La Compagnie des Glaces #08) - La Compagnie de la Banquise.epub
#
# the first date is inclusion of file in calibre.
# We will save it, suppress it from filename, and change modification time of book with this time.
#
# br, Jul 24, 2018

# first we find files corresponding to the pattern [0-9]{12}-.*\.epub in direct subdirectories,
# in case we added a single book, we do not want to do something stupid

find . -maxdepth 2 -regextype sed -type f -regex '\.\/[a-z-]*/*[0-9]\{12\}-.*\.epub$' -print | \
while read fullname
do
dirname=$(dirname "$fullname")
basename=$(basename "$fullname")
date=$(echo "$basename" | cut -c 1-12)
file=$(echo "$basename" | cut -c 14-)
newname="$dirname/$file"
mv "$fullname" "$newname"
touch -t $date "$newname"
echo processed "$fullname"
done

exit 0
----------------------------------------------------------------------

Next step, as I regular backup, I think I can find correct {last_modified} information. If the CLI Calibre permits to get/set this field, I believe I could restore everything to a normal status. I could also find information in some Calibre dates... In that case, I will change everything and choose the latest of meta-data and book last change.

[bodiccea]

-rw-rw-r-- 1 br br 1119387 2014-02-15 00:00 fra/Gallo,Max - Nice.epub


The file modofication date has been changed to something more correct, even if not perfect as I said in my previous mail

[BetterRed]

@bodiccea - I wondered if this might be of any use ==>> https://www.mobileread.com/forums/sh...47&postcount=7

BR

[chaley]

Originally Posted by bodiccea

I added a custom column {#brtime} of type date and built from other values.
Its template is "{timestamp:format_date(yyyyMMddhhmm)}". The issue is {timestamp} is book change, not metadata. For me changing metadata is changing book too.

My "Saving books to disk" template is (one subdir per book language).
{languages}/{#brtime}-{author_sort} - {series:|(| } {series_index:0>2s|#|) - }{title}

Why create a custom column? You can do the same thing in the save-to-disk template

{languages}/{timestamp:format_date(yyyyMMddhhmm)}-{author_sort} - {series:|(| } {series_index:0>2s|#|) - }{title}

You can also use the last_modified date, which is the last time the metadata for the book changed.

{languages}/{last_modified:format_date(yyyyMMddhhmm)}-{author_sort} - {series:|(| } {series_index:0>2s|#|) - }{title}

[bodiccea]

@BetterRed,

I don't think it is practical for 3,000 books with a GUI, even as I don't know the last modification date.

For now, I will try, if possible (all by CLI interface, not GUI), so it can be scriptable:
1) try to get the last_modified date from a correct backup (1 have 7 days/4 weeks/12 months/3 years backups). I believe a few months old one will be fine.
2) try to inject these dates in new Calibre DB for each book (making a list of the few ones which were not found for some reason, such as author/title change). I will have to manually change these errors.
3) replace #brtime with MAX({last_modified}, {timestamp}). The reason I created this new brtime field was to end-up with this: The filesystem date will show the real last change, metadata or book.

I believe I can write a function for point 3, never wrote one in Calibre.

I need to study the CLI interface first, not sure how powerful/easy it is...

Another maybe simpler solution: As I have file system backups. I know which books were added recently (look in ebooks.raoult.com, french section).
I save them somewhere else, delete the library and import a backup, where {last_modified} fields are correct, and add the new books.

In that case, just the point (3) will be the difficult task:
I will need a new plugin, but python is an easy language, just not sure how it integrates in columns.

br.

[bodiccea]

@Chaley,

The issue is the {last_modification} field is wrong (a mistake from me somewhere, don't know what,when, and why). So I need first to restore them.

I created a new column to be able later to eventually choose the MAX({last_modified}, {timestamp}) for that column, as indicated in my answer to BetterRed.
And this date will be shown in file system as modification date for books (my last bash script above does this).

br.

[BetterRed]

@bodiccea - a books last_modified column will be updated whenever you change any column that affects a book, but what's not obvious is that a modification in one book, say the spelling of a tag, can ripple through many books.

I found that the rippling changes were almost invariably caused by corrections to my errors, so I gave up on trying to make any use of the last_modified column. Instead I maintain a custom column - '#last-change'. This is pretty easy for me because I do all my metadata editing in the book list. So, I just tab into the cell and press '=', to set it to 'now'. If it's an error correction I don't update '#last_change'.

And, if I change the content of a book I can update '#last_change', IIRC changing the content of a book with an editor does not update 'last-modified'. The vast majority of my books are not commercial publications, so I do a lot of content editing.

BR

[BetterRed]

Originally Posted by bodiccea

And this date will be shown in file system as modification date for books (my last bash script above does this).

br.

This is where you confuse me: is the 'modification date' referred to above, the file systems last modified date, or the date you put into the save to disk file name, i.e. calibre's '[metadata] last_modified'?

BR

[bodiccea]

@BetterRed,

I want the file system modification date to be the MAX({last_modified}, {timestamp}).
Or, to be more clear: I must set my 3000 books modification date on file system automatically from calibre data.
So any GUI (finder or other) is not useable: I have to extract the dates from calibre automatically, and only a script (already done, look in a previous message) can do it.

So the date seen on web site will almost be correct.
I did read somewhere that some changes on boot itself (for instance adding a chapter, this kind of things) may change the {timestamp} field, so taking the MAX wil be the secure way.
If this {timestamp} ("date" column in GUI) always stays the same, and is the insertion book date in library, the MAX() will maybe not be necessary). This field is not really well documented.

br.

[theducks]

Originally Posted by bodiccea

@BetterRed,

I want the file system modification date to be the MAX({last_modified}, {timestamp}).
Or, to be more clear: I must set my 3000 books modification date on file system automatically from calibre data.
So any GUI (finder or other) is not useable: I have to extract the dates from calibre automatically, and only a script (already done, look in a previous message) can do it.

So the date seen on web site will almost be correct.
I did read somewhere that some changes on boot itself (for instance adding a chapter, this kind of things) may change the {timestamp} field, so taking the MAX wil be the secure way.
If this {timestamp} ("date" column in GUI) always stays the same, and is the insertion book date in library, the MAX() will maybe not be necessary). This field is not really well documented.

br.

{last_modified} has no clue of what the actual book files Modified date is.
It refers SOLELY to DB actions. It will record the time you ADD (a book)
eg. using Open_with to Edit the EPUB does not change the {last_modified} value. ( I keep a custom column for the last edited date which is manually set) changing that, also bumps the last_modified timestamp
So if you are more interested in EDITED Dates, create your own column.

[bodiccea]

@theducks,

I know last_modified is about metadata change only.
I am less clear about the "date" column. As said, if you make some changes on the book itself (I did read it somewhere it could, if you add a chapter or maybe other changes), this date could be changed too. But again, I am not sure if it is true or not. So using the MAX will be the "secure" way just in case of.

In any case, my main issue (initial question, 1st email of this thread) was to have the file system file modified date the same as the Calibre last_modified date. With or without the MAX function. This is done with my script, which runs once a day a 2:00 AM (it takes seconds to run), the library (changes only) itself is copied to server at 2:30 AM daily.
Don't forget my issue was that that my "last_modified" dates were all lost a few days ago, probably a mistake from me. This is way I changed temporarily the file system date to the "book creation date", which is better, but not good enough, as I think I can recover last_modified from a backup.