Allow defaultConfigOverwrite to replace httpd.conf

[gibizer]

For nova-api and nova-metadata we now allow to replace the whole
apache httpd.conf file. The replacement can be a template the same
way as the default file in templates/novaapi/config/httpd.conf is a
template.

This is expected to be only used for advanced troubleshooting and not
for customization for production use. Any permanent customization needs
to go through the CR interface (and potentially need an RFE to extend
such interface)

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gibizer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [gibizer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gibizer]

Measurements
Same setup as in PR #1074 but here instead of 3 nova-api pods with 2 procs each we overwrite the httpd.conf and used a single nova-api pod with 6 procs instead. Overall we had the same amount of WSGI procs.

load	result PR 1074	result with this PR
c25, t10	2.3 req/sec, no kill	2.45 req/sec, no kill
c50, t20	2.21 req/sec, no kill	2.29 req/sec, no kill
c75, t20	2.13 req/sec, no kill, 30 req error	1.52 req/sec, no kill, 75 req error
c100, t20	2.33 req/sec, galera kill, 360 req error	2.66 req/sec, galera kill, 855 req error (~50%)

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.