[FileType Plugin] GetFileName - Page 11

AnselmD · 02-19-2017, 03:16 AM

Quote:

Originally Posted by Terisa de morgan

Thank you very much, I was trying to avoid for time response, but it's really the safest way: semaphores.

For analysing, isn't it useful to add thread id and time to your logs?

Terisa de morgan · 02-19-2017, 07:22 AM

Hi,

This version should resolve @AnselmD's problem for sure (bug more or less..) and hopefully @giuseppe's too.

Terisa de morgan · 02-19-2017, 12:46 PM

Quote:

Originally Posted by AnselmD

For analysing, isn't it useful to add thread id and time to your logs?

Not really. As the shared variable is displayed at all the functions, the evolution (and loss of books) can be traced.

AnselmD · 02-20-2017, 11:00 AM

Quote:

Originally Posted by Terisa de morgan

Hi,

This version should resolve @AnselmD's problem for sure (bug more or less..) and hopefully @giuseppe's too.

Thank you very much @terisa,
this looks much better now, nevertheless rarely the database seems to be under stress (BusyError: BusyError: database is locked).
Miscellaneous -> "Max. simultaneous conversion/new Download/plugin jobs" was set to 4.

Code:

DEBUG: 2382.7 File:  pg131.epub
DEBUG: 2382.7 Updating column  #fileext for book id:  711  with value:  epub
DEBUG: 2383.0 Updating column  #filename for book id:  711  with value:  pg131
Added The Pilgrim's Progress from this world to that which is to come / Delivered under the simi
litude of a dream, by John Bunyan to db in: 3.2
DEBUG: 2385.8 File:  pg128.epub
DEBUG: 2385.8 Updating column  #fileext for book id:  712  with value:  epub
DEBUG: 2386.1 Updating column  #filename for book id:  712  with value:  pg128
Added The Arabian Nights Entertainments to db in: 3.1
DEBUG: 2389.1 File:  pg126.epub
DEBUG: 2389.1 Updating column  #fileext for book id:  713  with value:  epub
DEBUG: 2389.4 Updating column  #filename for book id:  713  with value:  pg126
Added The Poison Belt to db in: 3.1
DEBUG: 2392.3 File:  pg125.epub
DEBUG: 2392.3 Updating column  #fileext for book id:  714  with value:  epub
DEBUG: 2392.5 Updating column  #filename for book id:  714  with value:  pg125
Added A Girl of the Limberlost to db in: 3.2
DEBUG: 2395.4 File:  pg124.epub
DEBUG: 2395.4 Updating column  #fileext for book id:  715  with value:  epub
DEBUG: 2395.7 Updating column  #filename for book id:  715  with value:  pg124
Added Deuterocanonical Books of the Bible / Apocrypha to db in: 3.1
Running file type plugin GetFileName failed with traceback:
Traceback (most recent call last):
  File "site-packages/calibre/customize/ui.py", line 209, in run_plugins_on_postadd
  File "calibre_plugins.getfilename.__init__", line 154, in postadd
  File "calibre_plugins.getfilename.config", line 110, in get_library_config
  File "site-packages/calibre/library/__init__.py", line 11, in db
  File "site-packages/calibre/db/legacy.py", line 76, in __init__
  File "site-packages/calibre/db/legacy.py", line 48, in create_backend
  File "site-packages/calibre/db/backend.py", line 402, in __init__
  File "site-packages/calibre/db/backend.py", line 576, in initialize_custom_columns
  File "site-packages/calibre/db/backend.py", line 822, in execute
  File "src/cursor.c", line 231, in resetcursor
BusyError: BusyError: database is locked
Added At the Earth's Core to db in: 2.5
DEBUG: 2400.9 File:  pg120.epub
DEBUG: 2400.9 Updating column  #fileext for book id:  717  with value:  epub
DEBUG: 2401.3 Updating column  #filename for book id:  717  with value:  pg120
Added Treasure Island to db in: 3.2
DEBUG: 2404.4 File:  pg119.epub
DEBUG: 2404.4 Updating column  #fileext for book id:  718  with value:  epub
DEBUG: 2404.7 Updating column  #filename for book id:  718  with value:  pg119
Added A Tramp Abroad to db in: 3.3
DEBUG: 2407.6 File:  pg121.epub
DEBUG: 2407.6 Updating column  #fileext for book id:  719  with value:  epub
DEBUG: 2407.8 Updating column  #filename for book id:  719  with value:  pg121
Added Northanger Abbey to db in: 3.1
DEBUG: 2410.7 File:  pg122.epub
DEBUG: 2410.7 Updating column  #fileext for book id:  720  with value:  epub
DEBUG: 2411.1 Updating column  #filename for book id:  720  with value:  pg122
Added The Return of the Native to db in: 3.2
DEBUG: 2414.0 File:  pg118.epub
DEBUG: 2414.0 Updating column  #fileext for book id:  721  with value:  epub
DEBUG: 2414.3 Updating column  #filename for book id:  721  with value:  pg118
Added Big Dummy's Guide to the Internet to db in: 3.2

Code:

Added 832 books in 2736.0 seconds

This is 1 book each 3.2 seconds. I wouldn't expect this as stressful for the database. I do not additional access the database and I have only one instance of calibre running. I am using a hard drive for the calibre library with ext4 file system and a SSD as boot (and temporary) drive.

An overview which processes accesses the metadata.db

Code:

lsof |grep metadata.db
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
      Output information may be incomplete.
calibre-d 24110              xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110              xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110              xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110              xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
GUIPool   24110 13424        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
GUIPool   24110 13424        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
GUIPool   24110 13424        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
GUIPool   24110 13424        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
QXcbEvent 24110 24116        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
QXcbEvent 24110 24116        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
QXcbEvent 24110 24116        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
QXcbEvent 24110 24116        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
Listener  24110 24132        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Listener  24110 24132        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Listener  24110 24132        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Listener  24110 24132        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
Server    24110 24135        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Server    24110 24135        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Server    24110 24135        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Server    24110 24135        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
calibre-d 24110 24136        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 24136        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 24136        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 24136        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
DeviceMan 24110 24137        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
DeviceMan 24110 24137        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
DeviceMan 24110 24137        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
DeviceMan 24110 24137        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
CheckForU 24110 24138        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
CheckForU 24110 24138        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
CheckForU 24110 24138        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
CheckForU 24110 24138        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
MetadataB 24110 24139        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
MetadataB 24110 24139        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
MetadataB 24110 24139        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
MetadataB 24110 24139        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
Qt\x20bea 24110 24142        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Qt\x20bea 24110 24142        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Qt\x20bea 24110 24142        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
Qt\x20bea 24110 24142        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
calibre-d 24110 30823        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30823        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30823        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30823        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
calibre-d 24110 30824        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30824        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30824        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30824        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
calibre-d 24110 30825        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30825        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30825        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30825        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal
calibre-d 24110 30826        xyz   15uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30826        xyz   23uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30826        xyz   33uw     REG               8,98  1340416   88997890 /media/xyz/ebooks/testfilename8/metadata.db
calibre-d 24110 30826        xyz   34u      REG               8,98     6704   89000540 /media/xyz/ebooks/testfilename8/metadata.db-journal

Click image for larger version

Name: Screenshot from 2017-02-20 16-17-26.png
Views: 182
Size: 177.2 KB
ID: 155145

debuglog.7z.zip

Terisa de morgan · 02-20-2017, 12:16 PM

I cannot help in this, sorry. If there is a problem retrieving the configuration from the database, there is not a lot of things the plugin can do (aside of catching the exception in a most elegant way).

And the reason for stress? Sorry, I have no idea.

AnselmD · 02-20-2017, 02:02 PM

Quote:

Originally Posted by Terisa de morgan

I cannot help in this, sorry. If there is a problem retrieving the configuration from the database, there is not a lot of things the plugin can do (aside of catching the exception in a most elegant way).

And the reason for stress? Sorry, I have no idea.

Don't worry about it. Thank you very much for your help. Are you able to add file name and path to the log, if this exception occurs? If I know, which one fails, I can redo it it.

Terisa de morgan · 02-20-2017, 02:07 PM

Quote:

Originally Posted by AnselmD

Don't worry about it. Thank you very much for your help. Are you able to add file name and path to the log, if this exception occurs? If I know, which one fails, I can redo it it.

Yes, That's totally feasible, no problem with it. The other aspects, it's necessary a much deeper knowledge about calibre than I have. Perhaps something I'm doing in the plugin, but I don't know.

AnselmD · 02-20-2017, 02:15 PM

If some one is interested:

Miscellaneous -> "Max. simultaneous conversion/new Download/plugin jobs" was set to 1.
(Added 822 books in 2635.6 seconds)

Now I have 5 books failed getting file name and a "BusyError: database is locked".

The time to add a book was 2.8 seconds and higher if succeed and 2.3 to 2.6 seconds, if failed because the database is locked.

Failed:
<2.3 (0x)
2.3 (2x)
2.4 (1x)
2.5 (1x)
2.6 (1x)

Success:
2.7 (0x)
2.8 (1x)
2.9 (23x)
>= 3.0 (793x)

debuglog.7z.zip

AnselmD · 02-20-2017, 02:54 PM

Quote:

Originally Posted by Terisa de morgan

Yes, That's totally feasible, no problem with it. The other aspects, it's necessary a much deeper knowledge about calibre than I have. Perhaps something I'm doing in the plugin, but I don't know.

I was astonished, line 265 __init__.py
debug_print ("File: ", nom_fich)
was not reached. I thought the error happens at line 268ff (I have much lesser knowledge with calibre than you and I have a lack of knowledge in python too - so at these lines I think you do not have a chance to interact), but it happens
line 154:
library_config = cfg.get_library_config(db)

Isn't it possible, if the error occurs, to wait some time and to do it again?

BetterRed · 02-20-2017, 03:22 PM

Quote:

Originally Posted by kovidgoyal

The safest way to pass data between the calls is to use the filesystem. Create a temporary file, use the file locking in calibre.utils.lock.ExclusiveFile to write to the file from the run() method and read from it in the postadd() method.

Quote:

Originally Posted by Terisa de morgan

Thank you very much, I was trying to avoid for time response, but it's really the safest way: semaphores.

Assuming the plugin is using the file system to store transient data, as was suggested by Kovid Goyal then:

Users who add books in large batches could try using the CALIBRE_TEMP_DIR Environment variable to a give calibre a 'dedicated' directory for temporary storage, on an SSD if possible, exclude the directory from AV scans, indexing, compression, encryption etc etc.

BR

Terisa de morgan · 02-20-2017, 04:01 PM

Quote:

Originally Posted by AnselmD

I was astonished, line 265 __init__.py
debug_print ("File: ", nom_fich)
was not reached. I thought the error happens at line 268ff (I have much lesser knowledge with calibre than you and I have a lack of knowledge in python too - so at these lines I think you do not have a chance to interact), but it happens
line 154:
library_config = cfg.get_library_config(db)

Isn't it possible, if the error occurs, to wait some time and to do it again?

No, I don't like sleeps in a plugin which is doing which can be a massive job. It can be added but I prefer to skip the file, sorry.

Terisa de morgan · 02-20-2017, 04:02 PM

Quote:

Originally Posted by BetterRed

Assuming the plugin is using the file system to store transient data, as was suggested by Kovid Goyal then:

Users who add books in large batches could try using the CALIBRE_TEMP_DIR Environment variable to a give calibre a 'dedicated' directory for temporary storage, on an SSD if possible, exclude the directory from AV scans, indexing, compression, encryption etc etc.

BR

The problem, @BetterRed, is that the lock is not accesing to the temporary file but the database itself. Without storing the path, the plugin doesn't read that file.

kovidgoyal · 02-20-2017, 09:38 PM

@Terisa: From that traceback you appear to be creating a new db instance. Dont do that, instead use the db instance passed into the postadd method.

Terisa de morgan · 02-21-2017, 12:22 AM

@kovidgoyal, thank you. It was a mistake from the beginning of the plugin (and I wasn't aware of it because it was in a part of the plugin I don't look at from some time ago).

This plugin shouldn't have thar mistake. If that doesn't work, well...

AnselmD · 02-22-2017, 02:12 PM

Quote:

Originally Posted by Terisa de morgan

@kovidgoyal, thank you. It was a mistake from the beginning of the plugin (and I wasn't aware of it because it was in a part of the plugin I don't look at from some time ago).

This plugin shouldn't have thar mistake. If that doesn't work, well...

I did one error free run today

, I will do another tomorrow.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[FileType Plugin] YVES Bible Plugin	ClashTheBunny	Plugins	27	01-16-2023 01:25 AM
How to sort quickly books by filetype?	construsol	Calibre	7	05-24-2014 06:20 PM
Calibre associated with every filetype?!	sej7278	Calibre	14	08-09-2011 10:36 AM
Zip Filetype Plugin?	rsingley	Plugins	7	02-11-2011 05:11 PM
How to register a filetype?	plisken	iRex	2	09-28-2009 07:16 AM

02-20-2017, 12:16 PM	#155
Terisa de morgan Grand Sorcerer Posts: 6,233 Karma: 11768331 Join Date: Jun 2009 Location: Madrid, Spain Device: Kobo Clara/Aura One/Forma,XiaoMI 5, iPad, Huawei MediaPad, YotaPhone 2	I cannot help in this, sorry. If there is a problem retrieving the configuration from the database, there is not a lot of things the plugin can do (aside of catching the exception in a most elegant way). And the reason for stress? Sorry, I have no idea.

02-20-2017, 02:15 PM	#158
AnselmD Zealot Posts: 105 Karma: 10 Join Date: Oct 2013 Device: none	If some one is interested: Miscellaneous -> "Max. simultaneous conversion/new Download/plugin jobs" was set to 1. (Added 822 books in 2635.6 seconds) Now I have 5 books failed getting file name and a "BusyError: database is locked". The time to add a book was 2.8 seconds and higher if succeed and 2.3 to 2.6 seconds, if failed because the database is locked. Failed: <2.3 (0x) 2.3 (2x) 2.4 (1x) 2.5 (1x) 2.6 (1x) Success: 2.7 (0x) 2.8 (1x) 2.9 (23x) >= 3.0 (793x) debuglog.7z.zip

02-20-2017, 09:38 PM	#163
kovidgoyal creator of calibre Posts: 43,857 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@Terisa: From that traceback you appear to be creating a new db instance. Dont do that, instead use the db instance passed into the postadd method.

Advert

Advert