Calibre knows nothing about anything outside of its ecosystem, so you must make external-to-Calibre files visible to your plugin yourself.
For tiny packages (e.g. a useful python module that you need and perhaps found on Github), you could simply hard-copy it into your plugin's folder to make it a private copy.
Example: from .fuzzywuzzy import fuzz, string_processing, utils
For non-trivial packages, you might try (never needed it myself, so no guarantees) something like this:
import sys
sys.path.append('/path_to/folderhavingyourfile')
import yourfile
Personally, I would defer any imports until the user invokes an action in ui.py to avoid premature allocation and use of memory. In other words, do not import it at the top of ui.py. Make the user click a pushbutton or cause some other event that requires that the file be imported for the first time before continuing.
DaltonST