You might try and exclude the Library folder from your A/V Active scan (only).
The book files should have been scanned when you downloaded/imported them.
If you have a really big metadata.db, it is possible that is the issue.
Mine is 60MB and a bulk metadata change is almost instant.
What drive (type and location)?