Auto Import Folder: kickstart option?

[jeffery]

Is there a way to tell Calibre to process the files in the auto import folder?

-when files are added to the folder while Calibre is not running Calibre does not recognize they exist when Calibre is next started.

-If Calibre hangs during an import and you restart, it does not recognize the files that remain in the import folder.

either way I have to move them out and back before Calibre *sees* that they exist.

[kovidgoyal]

calibre adds all files in that folder on startup. If that is not happening for you then something else is going wrong. You can try running calibre in debug mode and see if you get anything useful.

[jeffery]

I am seeing this
calibre, version 3.27.0
ERROR: Unhandled exception: <b>SQLError</b>:SQLError: no such savepoint: _apsw-0

calibre 3.27 embedded-python: True is64bit: False
Windows-10-10.0.16299-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.16299')
Python 2.7.12+
Windows: ('10', '10.0.16299', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Extract ISBN (1, 4, 4) && KoboTouchExtended (3, 0, 2) && Modify ePub (1, 3, 13) && Kobo Books (1, 3, 0) && Kobo Utilities (2, 6, 0) && Prettify Cover 0.9.12 && Manage Series (1, 2, 8) && all recipes Recipe Reader (1, 0, 1) && Kindle covers (G) (0, 3, 0)
Traceback (most recent call last):
File "site-packages\calibre\gui2\auto_add.py", line 217, in add_to_db
File "site-packages\calibre\gui2\auto_add.py", line 261, in do_add
File "site-packages\calibre\gui2\library\models.py", line 345, in add_books
File "site-packages\calibre\db\legacy.py", line 231, in add_books
File "site-packages\calibre\db\cache.py", line 1597, in add_books
File "site-packages\calibre\db\cache.py", line 63, in call_func_with_lock
File "site-packages\calibre\db\cache.py", line 1567, in create_book_entry
File "site-packages\calibre\db\cache.py", line 1358, in set_metadata
File "site-packages\calibre\db\cache.py", line 252, in reload_from_db
SQLError: SQLError: no such savepoint: _apsw-0

[kovidgoyal]

That's an error writing to the database. That usually indicates something is messing with your calibre library folder. Typical culprits are anti-virus programs, file sync programs like dropbox or nuttong your calibre library on network share.

[jeffery]

I added the calibre library folder to the virus scanners exclusion list. I'll see if the problem goes away. I don't use drop box or any other online back up. the folder is located on a USB raid 5 array.

[jeffery]

ok no errors produced this run. but nothing imported from the import folder. nothing deleted from it either. no real activity seen on the drive. just a few random maintenance like accesses. There are 2,852 books in the folder ready to be imported.

calibre Debug log
calibre 3.27 embedded-python: True is64bit: False
Windows-10-10.0.16299-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.16299')
Python 2.7.12+
Windows: ('10', '10.0.16299', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Extract ISBN (1, 4, 4) && KoboTouchExtended (3, 0, 2) && Prettify Cover 0.9.12 && Modify ePub (1, 3, 13) && Kobo Utilities (2, 6, 0) && all recipes Recipe Reader (1, 0, 1) && Manage Series (1, 2, 8) && Kindle covers (G) (0, 3, 0) && Kobo Books (1, 3, 0)
calibre 3.27 embedded-python: True is64bit: False
Windows-10-10.0.16299-SP0 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '10', '10.0.16299')
Python 2.7.12+
Windows: ('10', '10.0.16299', 'SP0', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Extract ISBN (1, 4, 4) && KoboTouchExtended (3, 0, 2) && Prettify Cover 0.9.12 && Modify ePub (1, 3, 13) && Kobo Utilities (2, 6, 0) && all recipes Recipe Reader (1, 0, 1) && Manage Series (1, 2, 8) && Kindle covers (G) (0, 3, 0) && Kobo Books (1, 3, 0)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 56.4995121951 x 56.6530434783
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Showing splash screen...
[0.05] splash screen shown
[0.05] Initializing db...
[3.70] db initialized
[3.70] Constructing main UI...
DEBUG: 0.0 KoboUtilites::action.py - loading translations
DEBUG: 0.0 KoboUtilites::dialogs.py - loading translations
DEBUG: 0.0 KoboUtilites::action.py - loading translations
DEBUG: 5.4 No Kobo Touch, Glo or Mini appears to be connected
DEBUG: 5.4 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
DEBUG: 5.4 KoboUtilities:set_toolbar_button_tooltip - start: text='None'
DEBUG: 5.4 KoboUtilities:set_toolbar_button_tooltip - setting to text='Utilities to use with Kobo ereaders

Driver: KoboTouchExtended'
[9.42] main UI initialized...
[9.42] Hiding splash screen
[12.09] splash screen hidden
[12.09] Started up in 12.09 seconds with 119645 books

[BetterRed]

IIRC adding large numbers of books in a single 'gulp' can cause problems.

So, try again with fewer books in auto add folder. I suggest start with say 100, if that works try double that, if that works rinse and repeat.

BR

[kovidgoyal]

Right click the add books button and choose add books from folder, many books per folder and select the auto add folder

auto add is really not designed for bulk adding.

[jeffery]

actually I was turning to auto adding because after a week of trying to get these books imported I had given up on the add books button.

using add books, it just slowly trickled to a stop. since that method did not remove books from the folder it was importing from ( which is fine normally ) I was just repeating the process and not getting anywhere.

[kovidgoyal]

It'll probably go faster now that you have excluded the A/V. But if not, simply add in batches of a few hundred books at a time.

[jeffery]

OK so it's not that auto add is not working. And it's not that any conversion is taking place, that is turned off. it's that with a lot of books in the folder, it takes forever for Calibre to start the import process. I am guessing that it is cataloging all the files before attempting to import them. but it shouldn't take 6-8 hours to begin the process. I could look over the list by hand in under a minute and decide with ones were ebooks and which weren't.

This is how my non programmer brain sees it. I figure its a folder and the contents will change so... see a file. look at the extension, if it's a known extension. attempt to import it. if not skip it. try the next. each pass start over. but get to work, why sit around chewing your cud. if the user is stupid enough to keep dumping so many non eBook files in there that Calibre cannot get to a real ebook, then that is their fault (and probably not possible to do).

I don't understand why it's taking so long for things to ramp up. This is an external 5Gbps USB connected raid box (it's fully capable of maxing out the USB 3 port during reads). It also has lots of pretty activity lights so I can see when it's accessed and prior to the import process there is little going on. Calibre seems to truly be ignoring the folder. I assume it had pulled the contents of the folder when it was launched and is spending the 6-8 hours mulling over the list before beginning the import process. Calibre's CPU usage was only a couple percent during that time.

[DaltonST]

The Job Spy plug-in has 55 tools, one of which is called "Tickle Auto-Adder to Wake Up", which tickles your OS to signal Calibre's auto-add file watcher that something needs to be done. The probable issue is that your OS is not communicating events properly to the file watcher, or if it is, they are not getting through to their destination. Hence, the Tickler.



DaltonST

[theducks]

is there a reason for running 32 bit Calibre?

[jeffery]

it's the one that comes with the portable package.

[theducks]

Originally Posted by jeffery

it's the one that comes with the portable package.

Portable only comes in 32bit