Elaborating on theduck's answer because I had written most of it before I noticed that he posted: one way to do it is to create new library and move all the books to it. The books left behind are the ones that calibre doesn't know about. You do what you will with those books. You do this with calibre's "Copy to library - delete after copy". The steps:
- Stop calibre
- If your library is on a network folder, a NAS, or in some shared folder like OneDrive, Dropbox, or Google Drive then move it out. My guess is that one of the above is true, which explains why your library is borked. For more info see this FAQ.
- Start calibre. If necessary switch to the problematic library.
- Ensure that the Copy to library is available somewhere: on the toolbar, the context menu, or a menu.
- Create a new library somewhere with Library / Switch/create library / Create an empty library. Check the box "Copy structure ...". Make sure that "somewhere" is *not* on a NAS or cloud folder.
- Switch back to the old library and select all the books in that library
- Use Copy to library. Select the new library with (Delete after copy)
- Let it run.
- When it finishes calibre will think that the the current (original) library is empty. Exit calibre.
- Look through the original library's folders and clean up stuff.
- Start calibre and switch to the new library.
- Add the orphan books from the original library's folder with calibre's "Add books / Add from folders and subfolders". Say "Yes" when it asks if all formats in a folder are the same book. You probably want to look at the options in Preferences / Adding books / Adding actions tab before doing the Add.