How to best start db from scratch?

[theducks]

Originally Posted by manawydan

I am fed up.

It all worked so nicely and then Calibre started to freeze.

Windows 7 64bit, Fujitsu Lifebook laptop, Intel Core i5-3230M, 2,6GHz, 4GB RAM, 750GB HDD, Intel HD.
Calibre portable 1.33.

I copied the old library over but it was not possible to simply rename and use it as Calibre Library or to switch to it.

Well, I started to move folders into the actual library and asked Calibre to restore the db.
That worked.

Repeated it with another bunch and so on.
I removed settings folder first, not sure if this was necessary though.

But then it started to show error messages. Mainly something like "calibre.exe stopped" etc.

I even tried to run it with Zoomtext disabled. No help.
I am really stuck.
The hardware is ok, the OS is new, up until 30-40000 books it went ok.

I just don't get it.

Look at the title-folder shown at the top in your error message
Now look at the bottom line in the same message


This leads me to believe you have multi-os file system artifacts (been there.) 2 or more entries, varying only by case (not at all valid in XP, sometimes SHOWS in explorer, but not valid to call from a program (Calibre), which expects Windows case insensitive rules.

You may have accessed the Calibre Library by a Case sensitive OS (like Linux) or used a Network on a share/cloud, leading to multiple, case different files. Calibter expects only one of any extension to be in the folder

[manawydan]

Originally Posted by theducks

This leads me to believe you have multi-os file system artifacts (been there.) 2 or more entries, varying only by case (not at all valid in XP, sometimes SHOWS in explorer, but not valid to call from a program (Calibre), which expects Windows case insensitive rules.

You may have accessed the Calibre Library by a Case sensitive OS (like Linux) or used a Network on a share/cloud, leading to multiple, case different files. Calibter expects only one of any extension to be in the folder

Ok, but why .. *scratch head*

The whole library is from an Windows XP OS.
It has been backed-up to a external HD und from there been copied to Windows 7 64bit.

Why is it possible to restore the DB in the beginning (I have started it again, adding only a few files every time and so far it works - there are warnings but the restoring is finished "successful").

If case sensitivity is a major problem that causes Calibre to stop working why not from the beginning?
Do warnings/errors accumulate?

[BetterRed]

@manawydan - you won't like this suggestion, but... FWIW

There were some changes around the time calibre 1.0 was released that changed some things about file path/name lengths - they were shortened a bit. There have been a couple of instances of subsequent problems, including getting more than one file of the same type into a book folder. I'm wondering if your problems are another consequence of those changes - although the other problems were not calibre crashing.

My other thought is that 100,000 books seems to be quite large in the world of calibre libraries, I'm wondering if you've bumped into a practical limit on your configuration.

Calibre Portable will support multiple libraries. I think you said you had 100,000+ books. So create 11 similar libraries in the Calibre Portable folder (Library01-Library11). Add first 10,000 books into Library01, do some tests, if OK load second 10,000 books into Library02, repeat until done.

The Add is important as that will ensure the folder/file names conform to the current 'rules' under the post version 1.0 regime.

If you get to 100,000+ books in 11 libraries without any apparent problems then copy the books from the Library02 to Library01, do some tests; if OK then copy the books from the Library03 to Library01 repeat until done or not OK.

BR

[theducks]

I had forgotten the filename shortening change
and IIRC there was a bug in the first release that did leave unwanted remnants.
IIalsoRC, there was a cleanup tool that ran in the next release

[BetterRed]

@theducks - Kovid recently indicated he would look into putting something into Check Library to do any necessary file name adjustments, see https://www.mobileread.com/forums/sho...6&postcount=13

BR

[theducks]

Originally Posted by BetterRed

@theducks - Kovid recently indicated he would look into putting something into Check Library to do any necessary file name adjustments, see https://www.mobileread.com/forums/sho...6&postcount=13

BR

I think that is to cover conditions like this.
Some folk skip quite a few updates so it may be possible the original cleanup was skipped.
I know I had a mess from using the same Library on W7 and XP (and Ubuntu where I did not expect existing books to be touched. Obviously, I did something I should not have)

[BetterRed]

@theducks - I was one of those who ran into the long file name glitch, but I don't recall any clean-up tools, I'm sure I would remember if I'd used one - but age plays tricks with memory. In the 2-3 years I've been using calibre I can't recall any calibre 'clean-up/fix-up tools'

IIRC Kovid wrote that one of the reasons he is careful to avoid changes to the database - even to remove legacy redundancies - is that it would require some sort of fix-up which inevitably would result in a lot of support 'calls'. And he also said that db schema hadn't been changed for 2-3 years - which given the enhancements that have been made over that time is pretty remarkable

Like you I don't skip updates. If I'm away for a few, I will download and install each one, do a few tests and look at new features and bug fixes and then install the next one.

BR

[theducks]

Originally Posted by BetterRed

@theducks - I was one of those who ran into the long file name glitch, but I don't recall any clean-up tools, I'm sure I would remember if I'd used one - but age plays tricks with memory. In the 2-3 years I've been using calibre I can't recall any calibre 'clean-up/fix-up tools'

IIRC Kovid wrote that one of the reasons he is careful to avoid changes to the database - even to remove legacy redundancies - is that it would require some sort of fix-up which inevitably would result in a lot of support 'calls'. And he also said that db schema hadn't been changed for 2-3 years - which given the enhancements that have been made over that time is pretty remarkable

Like you I don't skip updates. If I'm away for a few, I will download and install each one, do a few tests and look at new features and bug fixes and then install the next one.

BR

BR
I don't think they were USER tools. Just like DB restructuring, they only fire off only when a new version install calls them

[BetterRed]

What would happen if one skipped that version. Would the next version not have the restructuring code, or at least tell the user to install the earlier version first.

I can think of some developers who might do that, but Kovid... Wash your brain out for even thinking such a thing

BR

[chaley]

The last change to the DB schema was in June 2011 when I added the author link column.

Originally Posted by BetterRed

What would happen if one skipped that version. Would the next version not have the restructuring code, or at least tell the user to install the earlier version first.

I can think of some developers who might do that, but Kovid... Wash your brain out for even thinking such a thing

What calibre is started, schema upgrades are applied one by one. They are never skipped.

FWIW: I don't think that there was any "upgrade" processing that went along with correcting the path length calculations in the new db. At least I don't see any evidence of it in the code. Existing libraries that break the rules are accepted as is.

I have a memory of some other post or debug report where Kovid suggested running a search/replace that adds a character to the title then running it again to remove it. This process corrected all the file names. It would certainly take a while to run on a large library, given that it would do renames of almost every folder and file in the library. Ahhh ... it is me misremembering the thread that BR referred to. Search & replace was not mentioned, but usng it should end up in the same place.