Improve Consumer IT failure validation
We expect document_not_found exceptions to come from elasticsearch whenever we send partial documents to elasticsearch. We consider this a normal part of operation, but the flink elasticsearch client disagrees. It throws an exception which terminates the pipeline.
Unfortunately, afaict, flink / elasticsearch do not expose any way to make decisions about what constitutes a failing request. There is a definition baked into the elasticsearch client, and flink operates directly off of that.
Introduce a total hack that wraps the BulkResponse.Listener and adjusts the responses prior to them being processed by flinks error handling. We mutate the responses array to make the responses we want to pass look like passes.
Additionally,
An event that was recently added to the consumer IT wasn't actually being run due to a misunderstanding of the end offset being provided. Update the consumer test to call out the requests it wants to process by name, and update the events to all have a meaningful meta.id property. Additionally update the assertion against the fetch failure side outputs to specify exactly which request id we expect to have failed, and in what way. These same events were also used in the UpdateElasticsearchEmitterTest, make a similar update there to fetch requests by id.
This exposed a problem with the wiremock templating. Namely that the wiremock integration with handlebars does not support changing the delimiters, but we are using a custom delimiter. In particular the integration layer only passes the data between the first '{{' and the last '}}' to the templating engine, this caused the $ from the first '${{' in each template to not be provided to the templating engine. Change the delimiter to '{{$' which ensures the $ doesn't get chopped and can distinguish handlebars templating from wikitext templates.
Change-Id: Ia560671b136d178eccd5eb740e8a67b060ded9dd