@kovid: I found a way to repeat it.
- Turn off authentication. The problem happens with authentication on, but this is simpler.
- Connect using OPDS. I am using CC but I imagine any OPDS client will work.
- Browse to "Title". Covers are visible.
- Browse back to the top category list.
- In calibre, using "Preferences" stop then start the content server. Do not restart calibre.
- In the OPDS client, browse to Title. All the covers are gone. The cache folder is now empty.
It appears that the mtimes dict is not being cleared when the server is stopped.