VED-991 Integrate DQ reporting into FHIR API and Batch

[dlzhry2nhs]

Summary

• Routine Change

Integrates the data quality reports into the batch processing system and the Create and Update endpoints of the Imms FHIR API.

Please refer to the ticket for test evidence of both the logs and the reports which are produced and sent to S3.

• Sets up infrastructure e.g. env vars so that reports can be sent to S3. Policies should already have been set up.
• Adds relevant calls and tests to both the Batch and FHIR API parts of the system.
• Finally, a minor tweak made to the validation model to tighten the datetime validation performed.

Reviews Required

• Dev

Review Checklist

ℹ️ This section is to be filled in by the reviewer.

• I have reviewed the changes in this PR and they fill all of the acceptance criteria of the ticket.
• If there were infrastructure, operational, or build changes, I have made sure there is sufficient evidence that the changes will work.
• If there were changes that are outside of the regular release processes e.g. account infrastructure to setup, manual setup for external API integrations, secrets to set, then I have checked that the developer has flagged this to the Tech Lead as release steps.
• I have checked that no Personal Identifiable Data (PID) is logged as part of the changes.

[github-actions]

This branch is working on a ticket in the NHS England VED JIRA Project. Here's a handy link to the ticket:

VED-991

[mfjarvis]

This hopefully won't be an issue, but we will send a duplicate report if we retry processing with a different file encoding

[dlzhry2nhs]

This Lambda is hard to follow and should definitely be refactored if there is ever some time.

However, one of the benefits of how it has been written is if it falls over for that reason, it only continues processing from where it left off. See line 134. Therefore, proposing to keep this as is.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
99.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud