as far as i can tell, every record i added to the queue was displayed to the user,
there was just one strange behavior worth mentioning:
after every call to url using
PHP Code:
self.browser.open_novisit()
the result_queue would be emptied, so i had to back it up before the call to the open_novisit and restore it after the call, like this:
PHP Code:
# Create a separate queue for downloaded metadata
downloaded_metadata_queue = Queue()
# Before calling open_novisit(), save downloaded metadata
while not result_queue.empty():
item = result_queue.get() #get also remove the item
downloaded_metadata_queue.put(item)
# call the URL
response = self.browser.open_novisit(request)
# Merge the downloaded metadata back to result_queue after the call
while not downloaded_metadata_queue.empty():
item = downloaded_metadata_queue.get()
result_queue.put(item)