|
putting aside the finding of interesting data ...
If, for example, a plugin assigns a value to mi.price, then the template engine can retrieve that value. Currently doing so requires using the raw_field function, but there is no overriding reason that normal template references couldn't work. The only issue is field-specific formatting, but given that the template engine cannot know what the field means, the only thing the engine can choose to do is produce a string.
Given the above, then it is easy to imagine 'plugboards' that massage raw data and store it into arbitrary metadata fields. We have something similar to that function in metadata search/replace, which handles type and is_multiple mismatches.
As far as I can see, the creation and maintenance of the screen maps describing where 'interesting' information is and how it is to be scraped is the hard part. Especially maintenance, given that the page formats change on a regular basis. I did something like this for regression testing of an application. It turned out that maintaining the testing scraper was as hard as maintaining the application.
|