diff --git a/_config.yml b/_config.yml index 0e1dd7381..49dfa31c3 100644 --- a/_config.yml +++ b/_config.yml @@ -78,6 +78,9 @@ collections: openvox-server_8x: output: true permalink: '/openvox-server/8.x/:path:output_ext' + openvox-server_9x: + output: true + permalink: '/openvox-server/9.x/:path:output_ext' openvoxdb_latest: output: true @@ -146,6 +149,11 @@ defaults: type: openvox-server_8x values: nav: openvox-server_8x + - scope: + path: '' + type: openvox-server_9x + values: + nav: openvox-server_9x - scope: path: '' diff --git a/_data/nav/openvox-server_9x.yml b/_data/nav/openvox-server_9x.yml new file mode 100644 index 000000000..c43f8f0e0 --- /dev/null +++ b/_data/nav/openvox-server_9x.yml @@ -0,0 +1,170 @@ +--- +- text: About OpenVox Server + link: services_puppetserver.html +- text: Release notes + link: release_notes.html +- text: Deprecated features + link: deprecated_features.html +- text: Installing and upgrading + items: + - text: Before you begin + link: install_pre.html + - text: Install OpenVox Server + link: install_from_packages.html + - text: What gets installed and where + link: install_what_and_where.html + - text: Upgrading OpenVox Server + link: upgrade_minor.html +- text: Configuring OpenVox Server + items: + - text: Configuring OpenVox Server + link: configuration.html + - text: 'Main OpenVox Server configurations: puppetserver.conf' + link: config_file_puppetserver.html + - text: 'Global settings: global.conf' + link: config_file_global.html + - text: 'Webserver service: webserver.conf' + link: config_file_webserver.html + - text: 'Web application mount points: web-routes.conf' + link: config_file_web-routes.html + - text: 'Access control: auth.conf' + link: config_file_auth.html + - text: 'Metrics services: metrics.conf' + link: config_file_metrics.html + - text: 'Log level and location: logback.xml' + link: config_file_logbackxml.html + - text: Advanced logging configuration + link: config_logging_advanced.html + - text: 'CA service: ca.conf' + link: config_file_ca.html + - text: OpenVox Server use of puppet.conf settings + link: puppet_conf_setting_diffs.html + - text: 'Configuring product-level interactions (optional): product.conf' + link: config_file_product.html +- text: Using and extending OpenVox Server + items: + - text: Using Ruby gems + link: gems.html + - text: Subcommands + link: subcommands.html + - text: Infrastructure CRL + link: infrastructure_crl.html + - text: Intermediate CA + link: intermediate_ca.html + - text: Renewing and regenerating certificates + link: certificate_renewal.html + - text: External SSL termination + link: external_ssl_termination.html + - text: Restarting OpenVox Server + link: restarting.html +- text: Tuning, troubleshooting, and known issues + items: + - text: Known issues + link: known_issues.html + - text: SSL problems with load-balanced PuppetDB servers ("Server Certificate Change" + error) + link: ssl_server_certificate_change_and_virtual_ips.html + - text: OpenVox Server tuning guide + link: tuning_guide.html + - text: Scaling OpenVox Server with compilers + link: scaling_puppet_server.html +- text: HTTP API + items: + - text: HTTP API overview + link: http_api_index.html + - text: PSON + link: pson.html + - text: Puppet v3 API + items: + - text: Catalog + link: http_catalog.html + - text: Node + link: http_node.html + - text: Facts + link: http_facts.html + - text: File bucket file + link: http_file_bucket_file.html + - text: File content + link: http_file_content.html + - text: File metadata + link: http_file_metadata.html + - text: Report + link: http_report.html + - text: Environments + link: http_environments.html + - text: Puppet v4 API + items: + - text: Catalog + link: puppet-api/v4/catalog.html + - text: CA v1 API + items: + - text: Certificate + link: ca-api/v1/http_certificate.html + - text: Certificate request + link: ca-api/v1/http_certificate_request.html + - text: Certificate status + link: ca-api/v1/http_certificate_status.html + - text: Certificate revocation list + link: ca-api/v1/http_certificate_revocation_list.html + - text: Certificate clean + link: ca-api/v1/http_certificate_clean.html + - text: Certificate expirations + link: ca-api/v1/http_certificate_expirations.html + - text: Certificate renewal + link: ca-api/v1/http_certificate_renewal.html + - text: Bulk certificate sign + link: ca-api/v1/http_certificate_sign.html + - text: Server-specific API + items: + - text: Environment classes + link: puppet-api/v3/environment_classes.html + - text: Environment modules + link: puppet-api/v3/environment_modules.html + - text: Environment transports + link: puppet-api/v3/environment_transports.html + - text: Compile endpoint + link: puppet-api/v3/compile.html + - text: Static file content + link: puppet-api/v3/static_file_content.html + - text: File content + link: puppet-api/v3/file_content.html + - text: Tasks overview + link: puppet-api/v3/tasks.html + - text: Tasks details + link: puppet-api/v3/task_detail.html + - text: Plans overview + link: puppet-api/v3/plans.html + - text: Plans details + link: puppet-api/v3/plan_detail.html + - text: Admin API + items: + - text: Environment cache + link: admin-api/v1/environment-cache.html + - text: JRuby pool + link: admin-api/v1/jruby-pool.html + - text: Status API + items: + - text: Services endpoint + link: status-api/v1/services.html + - text: Simple endpoint + link: status-api/v1/simple.html + - text: Metrics + items: + - text: Monitoring OpenVox Server metrics + link: puppet_server_metrics.html + - text: HTTP client metrics + link: http_client_metrics.html + - text: Applying metrics to improve performance + link: puppet_server_metrics_performance.html + - text: v1 metrics API + link: metrics-api/v1/metrics_api.html + - text: v2 (Jolokia) metrics API + link: metrics-api/v2/metrics_api.html +- text: Developer information + items: + - text: Developer debugging + link: dev_debugging.html + - text: Running from source + link: dev_running_from_source.html + - text: Tracing code events + link: dev_trace_func.html diff --git a/_data/nav_map.yml b/_data/nav_map.yml index 011273e4f..e45cde4b8 100644 --- a/_data/nav_map.yml +++ b/_data/nav_map.yml @@ -18,6 +18,10 @@ collections: openvox-server_latest|openvox-server_8x base: /openvox-server/latest/ +- nav_key: openvox-server_9x + collections: openvox-server_9x + base: /openvox-server/9.x/ + - nav_key: openvoxdb_8x collections: openvoxdb_latest|openvoxdb_8x base: /openvoxdb/latest/ diff --git a/_data/navigation.yml b/_data/navigation.yml index 6576fa864..6876fa529 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -6,7 +6,7 @@ collections: [openvox_latest, openvox_8x, openvox_9x] - title: OpenVox Server url: /openvox-server/latest/ - collections: [openvox-server_latest, openvox-server_8x] + collections: [openvox-server_latest, openvox-server_8x, openvox-server_9x] - title: OpenVoxDB url: /openvoxdb/latest/ collections: [openvoxdb_latest, openvoxdb_8x] diff --git a/_data/products.yml b/_data/products.yml index 5a8873f59..f5fa65c5c 100644 --- a/_data/products.yml +++ b/_data/products.yml @@ -73,6 +73,10 @@ openvox-server: label: OpenVox Server latest: 8x versions: + - id: 9x + label: "9.x" + collection: _openvox-server_9x + base: /openvox-server/9.x/ - id: 8x label: "8.x" collection: _openvox-server_8x diff --git a/docs/_openvox-server_8x/puppet-api/v3/file_content.markdown b/docs/_openvox-server_8x/puppet-api/v3/file_content.markdown index c9b1e157a..eaee6fd2a 100644 --- a/docs/_openvox-server_8x/puppet-api/v3/file_content.markdown +++ b/docs/_openvox-server_8x/puppet-api/v3/file_content.markdown @@ -7,9 +7,9 @@ The `file_content` endpoint returns contents of the specified file. ## `GET /puppet/v3/file_content/:mount_point/:module/:file-path?environment=:environment` -When specifying environment see the [OpenVox API docs](/openvox-server/latest/http_file_content.html) +When specifying environment see the [OpenVox API docs](/openvox-server/latest/http_file_content.html). -## `GET /puppet/v3/file_content/:mount_point/:module/:file-path?project=:project-ref +## `GET /puppet/v3/file_content/:mount_point/:module/:file-path?project=:project-ref` Return the contents of a file from the project specified by :project-ref, which should be of the form :name-:version diff --git a/docs/_openvox-server_9x/admin-api/v1/environment-cache.markdown b/docs/_openvox-server_9x/admin-api/v1/environment-cache.markdown new file mode 100644 index 000000000..db539f2e0 --- /dev/null +++ b/docs/_openvox-server_9x/admin-api/v1/environment-cache.markdown @@ -0,0 +1,36 @@ +--- +layout: default +title: "OpenVox Server: Admin API: Environment Cache" +--- + +When using directory environments, the OpenVox Server [caches](https://docs.openvoxproject.org/openvox/latest/environments_creating.html) the data it loads from disk for each environment. +OpenVox Server adds a new endpoint to the master's HTTP API: + +## `DELETE /puppet-admin-api/v1/environment-cache` + +To trigger a complete invalidation of the data in this cache, make an HTTP request to this endpoint. + +### Query Parameters + +This endpoint accepts an optional query parameter, `environment`, whose value may be set to the name of a specific OpenVox environment. If this parameter is provided, only the specified environment will be +flushed from the cache, as opposed to all environments. + +### Response + +A successful request to this endpoint will return an `HTTP 204: No Content`. The response body will be empty. + +### Example + +```text +$ curl -i --cert $(puppet config print hostcert) --key $(puppet config print hostprivkey) --cacert $(puppet config print localcacert) -X DELETE https://$(puppet config print certname):8140/puppet-admin-api/v1/environment-cache +HTTP/1.1 204 No Content + +$ curl -i --cert $(puppet config print hostcert) --key $(puppet config print hostprivkey) --cacert $(puppet config print localcacert) -X DELETE https://localhost:8140/puppet-admin-api/v1/environment-cache?environment=production +HTTP/1.1 204 No Content +``` + +## Relevant Configuration + +Access to this endpoint is controlled by the `puppet-admin` section of `puppetserver.conf`. See [the configuration page](../../configuration.html) for more information. + +In the example above, the `curl` command is using a certificate and private key. You must make sure this certificate's name is included in the `puppet-admin -> client-whitelist` setting before you can use it. diff --git a/docs/_openvox-server_9x/admin-api/v1/jruby-pool.markdown b/docs/_openvox-server_9x/admin-api/v1/jruby-pool.markdown new file mode 100644 index 000000000..b2e392db7 --- /dev/null +++ b/docs/_openvox-server_9x/admin-api/v1/jruby-pool.markdown @@ -0,0 +1,66 @@ +--- +layout: default +title: "OpenVox Server: Admin API: JRuby Pool" +--- + +OpenVox Server contains a pool of JRuby instances. OpenVox Server adds a new, experimental endpoint to the master's HTTP API: + +## `DELETE /puppet-admin-api/v1/jruby-pool` + +This will remove all of the existing JRuby interpreters from the pool, allowing the memory occupied by these interpreters to be reclaimed by the JVM's garbage collector. The pool will then be refilled with new +JRuby instances, each of which will load the latest Ruby code and related resources from disk. + +If you're developing new Ruby plugins that run on the OpenVox server (functions, resource types, report handlers), you may need to force OpenVox to re-load its plugins when a new version is ready to test. Killing +the JRuby instances will do this, and it's faster than restarting the entire JVM process. + +Furthermore, if you are using multiple environments, this could be useful if you want to make sure that your JRuby instances are cleaned up and don't have conflicts based on common code that appears in multiple +environments. + +This is an experimental feature, and as such the performance impact is unknown at this time. Also, please note that this operation is computationally expensive, and as such OpenVox Server will be unable to +fulfill any incoming requests until the first of the new interpreters has been initialized, which may take several seconds. + +### Response + +A successful request to this endpoint will return an `HTTP 204: No Content`. The response body will be empty. + +### Example + +```text +$ curl -i --cert $(puppet config print hostcert) --key $(puppet config print hostprivkey) --cacert $(puppet config print localcacert) -X DELETE https://$(puppet config print certname):8140 +/puppet-admin-api/v1/jruby-pool +HTTP/1.1 204 No Content +``` + +## `GET /puppet-admin-api/v1/jruby-pool/thread-dump` + +Retrieve a Ruby thread dump for each JRuby instance registered to the pool. The thread dump provides a backtrace through the Ruby code that each instance is executing and is useful for diagnosing instances that +have stalled or are otherwise unresponsive. Backtraces are generated using the JRuby JMX interface and require the `jruby.management.enabled` property to be set to `true` in the JVM running OpenVox Server. + +### Response + +A successful request to this endpoint will return a `HTTP 200: Ok` status code. The response body will be a JSON document containing a map that associates each JRuby instance ID with a map containing a +`thread-dump` entry that has a string value with the Ruby backtrace. + +A `HTTP 500: Internal Server Error` status code will be returned if an exception occurs while retrieving the thread dump for a JRuby instance, or if the `jruby.management.enabled` property is not set to `true`. +The response body in this case is also JSON, but the failed instances will be associated with a map containing a `error` entry with a value describing the issue. + +### Example + +```text +$ curl -i --cert $(puppet config print hostcert) --key $(puppet config print hostprivkey) --cacert $(puppet config print localcacert) -X GET https://$(puppet config print certname):8140/puppet-admin-api/v1/jruby-pool/thread-dump +HTTP/1.1 200 OK + +{"1":{"thread-dump":"All threads known to Ruby instance 1960016402\n\n ..."}} + +# Error returned when jruby.management.enabled is not configured +$ curl -i --cert $(puppet config print hostcert) --key $(puppet config print hostprivkey) --cacert $(puppet config print localcacert) -X GET https://$(puppet config print certname):8140/puppet-admin-api/v1/jruby-pool/thread-dump +HTTP/1.1 500 Server Error + +{"1":{"error":"JRuby management interface not enabled. Add '-Djruby.management.enabled=true' to JAVA_ARGS to enable thread dumps."}} +``` + +## Relevant Configuration + +Access to this endpoint is controlled by the `puppet-admin` section of `puppetserver.conf`. See [the configuration page](../../configuration.html) for more information. + +In the example above, the `curl` command is using a certificate and private key. You must make sure this certificate's name is included in the `puppet-admin -> client-whitelist` setting before you can use it. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate.md new file mode 100644 index 000000000..1d52b09f4 --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate.md @@ -0,0 +1,106 @@ +--- +layout: default +title: 'OpenVox Server CA API: Certificate' +--- + +# Certificate + +The `certificate` endpoint returns the certificate for the specified name, +which might be either a standard certname or `ca`. + +## Find + +Get a certificate. + + GET /puppet-ca/v1/certificate/:nodename + + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`text/plain` + +The returned certificate is always in the `.pem` format. + +### Parameters + +None + +### Responses + +#### Certificate found + + GET /puppet-ca/v1/certificate/elmo.mydomain.com + + HTTP 200 OK + Content-Type: text/plain + + -----BEGIN CERTIFICATE----- + MIIFujCCA6KgAwIBAgIBATANBgkqhkiG9w0BAQsFADBiMWAwXgYDVQQDDFdQdXBw + ZXQgQ0EgZ2VuZXJhdGVkIG9uIGRoY3A1MC5reWxvLmJhY2tsaW5lLnB1cHBldGxh + YnMubmV0IGF0IDIwMTMtMDYtMjQgMTY6MzA6MTcgLTA3MDAwHhcNMTMwNjIzMjMz + MDE5WhcNMTgwNjIzMjMzMDE5WjBiMWAwXgYDVQQDDFdQdXBwZXQgQ0EgZ2VuZXJh + dGVkIG9uIGRoY3A1MC5reWxvLmJhY2tsaW5lLnB1cHBldGxhYnMubmV0IGF0IDIw + MTMtMDYtMjQgMTY6MzA6MTcgLTA3MDAwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw + ggIKAoICAQDABq1lmzccjuRmnCdXvTmdeXJGb9S8r8+I+G6fkHTa1WKDSob9PZpS + eXJtanbl0zNws9yBt1Dko2zhKDKctBRWf5CT42nDxBZPY7SaD7KaCzb07g9wfWgU + BOb/6smyl/iySEmQzzFLRgZbo5A9WLiy/UdyQim1faakevRme2Xi/l/i0TKbpu27 + DhCS+E8aC8Bvaj0ph0T+TzYphTR76pP5Kps6G7Jyk/HFYrVXnY44X2PEt2mgkEXp + xHCbU+qCFMtTLMG+ZArA/noM3I/O6W5LhLSzApjut/M7UdMlpZ45PGDrsvf2R306 + NcOh+zbbkhxuIaGqaxeaenYzbOlA3gXhZvYaV6EKjXNtm7BslpsvhLi0U+CWyb3C + qRkpex0MgxJgxoqViJ4TDVA+EmztOnK86+G4HGeJqTPQloYO/Td1wMT1Txh9T5Ue + Wctw/g+4o22EyJQRo+vxxzHNRIfe7EHAerMUtLT5u9MJeQb9N1iUR2ATNAN+QiB2 + KEqyc9eMapK6QUZFV23Xvbdup1WCrgsWXBqyRWKV7x0sc9Wv8RMRKEFYaBeHEVXU + m0hGgF34Z8Rzphq2H1FjkLD+xbtGOjrA1Mb2De81Hfvrf18497X5UMPtsuzOt/XU + PHbbSCy+05J7VNZ/gaiGqgpHfcG5yiqCdj1LIzhFuuvm+fADPxK38wIDAQABo3sw + eTA3BglghkgBhvhCAQ0EKhYoUHVwcGV0IFJ1YnkvT3BlblNTTCBJbnRlcm5hbCBD + ZXJ0aWZpY2F0ZTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV + HQ4EFgQUEhn/MqSDtuxg12klWosCGenxf1cwDQYJKoZIhvcNAQELBQADggIBAH1G + L3FG/keKlGqs70PxxvR1wCo4VM3K/C+5uxnzm1MHEAd96nhtwE6YSkUe+XgDiXfC + +NXS2C4TeTQAEo6grREapWDjhJvrhrgqTZmb4lTKzb91II3/VGYzG5UXxID262zy + QLoX/IBN/xDJ5ds0wF2adUbnHUssEGGljgngewH/7kjeW/L5iL+USXZnKHPSggjM + RAEjlucE/rDqDNoxhOS4K2PjseFm7krW4cZ0gNmxdrhc7OhmJ56dH92F4M9jn7Qy + EqxWB304U/aMcO3NJxTQc7AreL/pUtjtI6hxM4miHbjSh6RfNBqhzRyJvxA6gc6g + m3kumdw04KZFSs/6fPFFbI60i5K+vioB4CnUWpj+3Z+OnDEvhQJEACR1JC8A67Ih + x+GDlbHLU1BWonwZzSMJz+ABXV3dwIrOSFHI0UmDXg+cIdZ+SaL93qMjUVU4v9nu + gR9yJGMqNuzLjgfbD/KGCEEAITKBwPvCVd//OMlWVrXr7vvt+yo6STIlTJxABJDp + CSLyHUtT++CsPXsPADxgRctpIbh1eMFEivkK9Oy+W/CZYIZnARVysUpMWg7TkXqx + mSCXy9ZXLWqU/ssVhbLS9vFVa5pvxcyfiRpsFg0XZsx8mnZP6OaWcL8FjF+/NwNP + tg1+DuYTn+d54OHi/GZEnvutgrDZyrJDrrb/Czm9 + -----END CERTIFICATE----- + +#### Certificate not found + + GET /puppet-ca/v1/certificate/certificate_does_not_exist + + HTTP 404 Not Found + Content-Type: text/plain + + Not Found: Could not find certificate certificate_does_not_exist + +#### No Certificate name given + + GET /puppet-ca/v1/certificate + + HTTP/1.1 400 Bad Request + Content-Type: text/plain + + No request key specified in /puppet-ca/v1/certificate + +#### Master is not a CA + + GET /puppet/v1/certificate/valid_certificate + + HTTP/1.1 400 Bad Request + Content-Type: text/plain + + this master is not a CA + + +## Schema + +A `certificate` response body is not structured data according to any standard scheme such as +json/pson/yaml, so no schema is applicable. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_clean.markdown b/docs/_openvox-server_9x/ca-api/v1/http_certificate_clean.markdown new file mode 100644 index 000000000..8d75efc1e --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_clean.markdown @@ -0,0 +1,49 @@ +--- +layout: default +title: "OpenVox Server CA API: Certificate Clean" +--- +# Certificate Clean + +The `certificate clean` endpoint of the CA API allows the user to revoke and delete a list +of certificates with a single request. + +```http +PUT /puppet-ca/v1/clean +Content-Type: application/json +``` + +The body takes one required key, `certnames`, a list of certificates to clean. Each cert +in the list will be revoked, then the associated certficate file will be deleted from the CA. + +If a given certname does not have an associated signed cert on the CA, the response body will +call this out, but the request will not error. + +## Example + +```http +PUT /puppet-ca/v1/clean +Content-Type: application/json +Content-Length: 58 + +{"certnames":["agent1.example.net","agent2.example.net"]} + +HTTP/1.1 200 OK +Context-Type: text/plain +Successfully cleaned all certificates. +``` + +Both certs will be revoked, then have their files deleted. + +```http +PUT /puppet-ca/v1/clean +Content-Type: application/json +Content-Length: 58 + +{"certnames":["missing.example.net","agent1.example.net"]} + +HTTP/1.1 200 OK +Context-Type: text/plain +The following certs do not exist and cannot be revoked: ["missing.example.net"] +``` + +The missing cert is skipped, the other is revoked and deleted. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_expirations.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate_expirations.md new file mode 100644 index 000000000..eca2ade05 --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_expirations.md @@ -0,0 +1,59 @@ +--- +layout: default +title: "OpenVox Server CA API: Certificate Expirations" +--- + +## Certificate Expirations + +The `expirations` endpoint returns the `not-after` date for every certificate in the CA bundle and the +`next-update` date of every CRL in the chain. This is useful for monitoring CA infrastructure health and +alerting before CA certificate or CRL expiration causes service disruption. + +## Get + +Retrieve expiration information for the CA bundle and CRL chain. + + GET /puppet-ca/v1/expirations + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`application/json` + +### Parameters + +None + +### Access + +Any client with a certificate signed by the CA may access this endpoint. Access is controlled by OpenVox +Server's `ca.conf` authorization configuration. + +### Example Response + + GET /puppet-ca/v1/expirations + + HTTP/1.1 200 OK + Content-Type: application/json + + { + "ca-certs": { + "Puppet CA: ca.example.com": "2041-05-15T11:22:16UTC", + "Puppet Root CA: a3f8c1d74b2e05": "2041-05-15T11:22:14UTC" + }, + "crls": { + "Puppet CA: ca.example.com": "2031-05-26T14:36:05UTC", + "Puppet Root CA: a3f8c1d74b2e05": "2041-05-15T11:22:14UTC" + } + } + +The response contains two top-level keys: + +- `ca-certs` — maps each CA certificate's subject name to its `not-after` date. +- `crls` — maps each CRL issuer's subject name to its `next-update` date. + +Timestamps are in UTC. The subject name keys reflect the CN of each certificate in the CA bundle. +Deployments using an intermediate CA will have one entry per CA in the chain, as shown above. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_renewal.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate_renewal.md new file mode 100644 index 000000000..ec5454329 --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_renewal.md @@ -0,0 +1,88 @@ +--- +layout: default +title: "OpenVox Server CA API: Certificate Renewal" +--- + +## Certificate Renewal + +The `certificate_renewal` endpoint allows an OpenVox agent to automatically renew its own certificate without +requiring an administrator to sign a new CSR. The agent makes the request authenticated with its current +certificate (via mutual TLS), and if auto-renewal is enabled and the certificate was signed by this CA, a new +certificate is returned. + +> **Note:** This endpoint is disabled by default. It requires `certificate-authority.allow-auto-renewal: true` +> in OpenVox Server's CA configuration before it will return anything other than 404. + +## Renew + +Submit a certificate renewal request. + + POST /puppet-ca/v1/certificate_renewal + +The request has no body. OpenVox Server identifies the certificate to renew from the TLS client certificate +presented during the connection. If `allow-header-cert-info` is enabled in the CA settings, the certificate +may instead be provided via the `X-Client-Cert` HTTP header. + +### Supported HTTP Methods + +POST + +### Supported Response Formats + +`text/plain` (PEM-encoded signed certificate) + +### Parameters + +None + +### Configuration + +Two settings in the `certificate-authority` section of OpenVox Server's configuration control this endpoint: + +- `allow-auto-renewal` — must be `true` to enable the endpoint. Defaults to `false`. +- `auto-renewal-cert-ttl` — sets the validity period of the renewed certificate. Defaults to `90d`. + +### Access + +Any client with a certificate signed by the CA may access this endpoint. Access is controlled by OpenVox +Server's `ca.conf` authorization configuration. The endpoint should never be used unauthenticated, as it +requires a valid client certificate to identify the certificate to renew. + +### Responses + +#### Renewal successful + + POST /puppet-ca/v1/certificate_renewal + + HTTP/1.1 200 OK + Content-Type: text/plain + + -----BEGIN CERTIFICATE----- + ... (new signed certificate) ... + -----END CERTIFICATE----- + +The renewed certificate is valid for the period configured in `auto-renewal-cert-ttl`. + +#### Auto-renewal disabled + + HTTP/1.1 404 Not Found + +`certificate-authority.allow-auto-renewal` is not enabled. + +#### Certificate not recognized + + HTTP/1.1 403 Forbidden + Content-Type: text/plain + + Certificate present, but does not match signature + +The certificate presented was not signed by this CA. + +#### No valid certificate in request + + HTTP/1.1 400 Bad Request + Content-Type: text/plain + + No valid certificate found in renewal request + +No client certificate was found in the TLS session or the `X-Client-Cert` header. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_request.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate_request.md new file mode 100644 index 000000000..efdc9de2a --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_request.md @@ -0,0 +1,165 @@ +--- +layout: default +title: 'OpenVox Server CA API: Certificate Request' +--- + +# Certificate Request + +The `certificate_request` endpoint submits a Certificate Signing Request (CSR) +to the master. The master must be configured to be a CA. The returned +CSR is always in the `.pem` format. + +## Find + +Get a submitted CSR + + GET /puppet-ca/v1/certificate_request/:nodename + Accept: text/plain + +## Save + +Submit a CSR + + PUT /puppet-ca/v1/certificate_request/:nodename + Content-Type: text/plain + +Note: The `:nodename` must match the Common Name on the submitted CSR. + +Note: Although the `Content-Type` is sent as `text/plain` the content is +specifically a CSR in PEM format. + +## Search + +**Note:** The plural `certificate_requests` endpoint is a legacy feature. Puppet +Server doesn't support it, and we don't plan to add support in the future. + +List submitted CSRs + + GET /puppet-ca/v1/certificate_requests/:ignored_pattern + Accept: text/plain + +The `:ignored_pattern` parameter is not used, but must still be provided. + +## Destroy + +Delete a submitted CSR + + DELETE /puppet-ca/v1/certificate_request/:nodename + Accept: text/plain + +### Supported HTTP Methods + +The default configuration only allows requests that result in a Find and a +Save. You need to modify auth.conf in order to allow clients to use Search and +Destroy actions. It is not recommended that you change the default settings. + +GET, PUT, DELETE + +### Supported Response Formats + +`text/plain` + +The returned CSR is always in the `.pem` format. + +### Parameters + +None + +### Examples + +#### CSR found + + GET /puppet-ca/v1/certificate_request/agency + + HTTP/1.1 200 OK + Content-Type: text/plain + + -----BEGIN CERTIFICATE REQUEST----- + MIIBnzCCAQwCAQAwYzELMAkGA1UEBhMCVUsxDzANBgNVBAgTBkxvbmRvbjEPMA0G + A1UEBxMGTG9uZG9uMSEwHwYDVQQKExhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQx + DzANBgNVBAMTBmFnZW5jeTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxSCr + FKUKjVGFPuQ0iGM9mZKw94sOIgGohqrHH743kPvjsId3d38Qk+H+1DbVf42bQY0W + kAVcwNDqmBnx0lOtQ0oeGnbbwlJFjhqXr8jFEljPrc9S2/IIILDf/FeYWw9lRiOV + LoU6ZfCIBfq6v4D4KX3utRbOoELNyBeT6VA1ufMCAwEAAaAAMAkGBSsOAwIPBQAD + gYEAno7O1jkR56TNMe1Cw/eyQUIaniG22+0kmoftjlcMYZ/IKCOz+HRgnDtBPf8j + O5nt0PQN8YClW7Xx2U8ZTvBXn/UEKMtCBkbF+SULiayxPgfyKy/axinfutEChnHS + ZtUMUBLlh+gGFqOuH69979SJ2QmQC6FNomTkYI7FOHD/TG0= + -----END CERTIFICATE REQUEST----- + +#### CSR not found + + GET /puppet-ca/v1/certificate_request/does_not_exist + + HTTP/1.1 404 Not Found + Content-Type: text/plain + + Not Found: Could not find certificate_request does_not_exist + +#### No node name given + + GET /puppet-ca/v1/certificate_request + + HTTP/1.1 400 Bad Request + Content-Type: text/plain + + No request key specified in /puppet-ca/v1/certificate_request + +#### Delete a CSR that exists + + DELETE /puppet-ca/v1/certificate_request/agency + Accept: s + + HTTP/1.1 200 OK + Content-Type: text/plain + + 1 + +#### Delete a CSR that does not exists + + DELETE /puppet-ca/v1/certificate_request/missing + Accept: s + + HTTP/1.1 200 OK + Content-Type: text/plain + + false + +#### Retrieve all CSRs + + GET /puppet-ca/v1/certificate_requests/ignored + Accept: s + + HTTP/1.1 200 OK + Content-Type: text/plain + + -----BEGIN CERTIFICATE REQUEST----- + MIIBnzCCAQwCAQAwYzELMAkGA1UEBhMCVUsxDzANBgNVBAgTBkxvbmRvbjEPMA0G + A1UEBxMGTG9uZG9uMSEwHwYDVQQKExhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQx + DzANBgNVBAMTBmFnZW5jeTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxSCr + FKUKjVGFPuQ0iGM9mZKw94sOIgGohqrHH743kPvjsId3d38Qk+H+1DbVf42bQY0W + kAVcwNDqmBnx0lOtQ0oeGnbbwlJFjhqXr8jFEljPrc9S2/IIILDf/FeYWw9lRiOV + LoU6ZfCIBfq6v4D4KX3utRbOoELNyBeT6VA1ufMCAwEAAaAAMAkGBSsOAwIPBQAD + gYEAno7O1jkR56TNMe1Cw/eyQUIaniG22+0kmoftjlcMYZ/IKCOz+HRgnDtBPf8j + O5nt0PQN8YClW7Xx2U8ZTvBXn/UEKMtCBkbF+SULiayxPgfyKy/axinfutEChnHS + ZtUMUBLlh+gGFqOuH69979SJ2QmQC6FNomTkYI7FOHD/TG0= + -----END CERTIFICATE REQUEST----- + + --- + -----BEGIN CERTIFICATE REQUEST----- + MIIBnjCCAQsCAQAwYjELMAkGA1UEBhMCVUsxDzANBgNVBAgTBkxvbmRvbjEPMA0G + A1UEBxMGTG9uZG9uMSEwHwYDVQQKExhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQx + DjAMBgNVBAMTBWFnZW50MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC1tucK + enT1CkDPgsCU/0e2cbzRsiKF8yHH7+ntF6Q3d9ZCaZWJ00mj0+YmiYrnum+KAikE + 45Iaf9vaUV3CPsDVrUPOI8kYehiv868ZhP3nxblE6iuNBK+Fdv9GN/vKQrmL5iRE + bIrOM3/lxpS7SpidGdA6EIVlS3604bwLY4xHNQIDAQABoAAwCQYFKw4DAg8FAAOB + gQAXH0YFuidPqB6P2MyPEEGZ3rzozINBx/oXvGptXI60Zy5mgH6iAkrZfi57pEzP + jFoO2JRaFxTJC1FVpc4zR1K6sq4h3fIMwqppJRX+5wJNKyhU61eY2gR2O/rAJzw4 + wcUKf9JhoE7/p1cUulIIIq7t/ibCvf0LYSFwGqTwGqN2TQ== + -----END CERTIFICATE REQUEST----- + +The CSR PEMs are separated by "\n---\n" + +## Schema + +A `certificate_request` response body is not structured data according to any +standard scheme such as json/pson/yaml, so no schema is applicable. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_revocation_list.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate_revocation_list.md new file mode 100644 index 000000000..d0da8d694 --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_revocation_list.md @@ -0,0 +1,191 @@ +--- +layout: default +title: 'OpenVox Server CA API: Certificate Revocation List' +--- + +# Certificate Revocation List + +The `certificate_revocation_list` endpoint retrieves a Certificate Revocation List (CRL) +from the master. The master must be configured to be a CA. The returned +CRL is always in the `.pem` format. + +The `:nodename` should always be `ca`, due to the default auth.conf rules for +WEBrick and Rack Puppet masters. (You can use a different `:nodename` if you +change the auth rules, but it will have no effect on the response.) + +## Find + +Get the submitted CRL + + GET /puppet-ca/v1/certificate_revocation_list/:nodename + Accept: text/plain + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`text/plain` + +The returned CRL is always in the `.pem` format. + +### Parameters + +None + +### Examples + +Because the returned CRL always looks similar to the human eye, the successful examples are each followed by an openssl +decoding of the CRL PEM file. + +#### Empty revocation list + + GET /puppet-ca/v1/certificate_revocation_list/ca + + HTTP/1.1 200 OK + Content-Type: text/plain + + -----BEGIN X509 CRL----- + MIICdzBhAgEBMA0GCSqGSIb3DQEBBQUAMB8xHTAbBgNVBAMMFFB1cHBldCBDQTog + bG9jYWxob3N0Fw0xMzA3MTYyMDQ4NDJaFw0xODA3MTUyMDQ4NDNaoA4wDDAKBgNV + HRQEAwIBADANBgkqhkiG9w0BAQUFAAOCAgEAqyBJOy3dtCOcrb0Fu7ZOOiDQnarg + IzXUV/ug1dauPEVyURLNNr+CJrr89QZnU/71lqgpWTN/J47mO/lffMSPjmINE+ng + XzOffm0qCG2+gNyaOBOdEmQTLdHPIXvcm7T+wEqc7XFW2tjEdpEubZgweruU/+DB + RX6/PhFbalQ0bKcMeFLzLAD4mmtBaQCJISmUUFWx1pyCS6pgBtQ1bNy3PJPN2PNW + YpDf3DNZ16vrAJ4a4SzXLXCoONw0MGxZcS6/hctJ75Vz+dTMrArKwckytWgQS/5e + c/1/wlMZn4xlho+EcIPMPfCB5hW1qzGU2WjUakTVxzF4goamnfFuKbHKEoXVOo9C + 3dEQ9un4Uyd1xHxj8WvQck79In5/S2l9hdqp4eud4BaYB6tNRKxlUntSCvCNriR2 + wrDNsMuQ5+KJReG51vM0OzzKmlScgIHaqbVeNFZI9X6TpsO2bLEZX2xyqKw4xrre + OIEZRoJrmX3VQ/4u9hj14Qbt72/khYo6z/Fckc5zVD+dW4fjP2ztVTSPzBqIK3+H + zAgewYW6cJ6Aan8GSl3IfRqj6WlOubWj8Gr1U0dOE7SkBX6w/X61uqsHrOyg/E/Z + 0Wcz/V+W5iZxa4Spm0x4sfpNzf/bNmjTe4M2MXyn/hXx5MdHf/HZdhOs/lzwKUGL + kEwcy38d6hYtUjs= + -----END X509 CRL----- + + > openssl crl -inform PEM -in empty.crl -text -noout + Certificate Revocation List (CRL): + Version 2 (0x1) + Signature Algorithm: sha1WithRSAEncryption + Issuer: /CN=Puppet CA: localhost + Last Update: Jul 16 20:48:42 2013 GMT + Next Update: Jul 15 20:48:43 2018 GMT + CRL extensions: + X509v3 CRL Number: + 0 + No Revoked Certificates. + Signature Algorithm: sha1WithRSAEncryption + ab:20:49:3b:2d:dd:b4:23:9c:ad:bd:05:bb:b6:4e:3a:20:d0: + 9d:aa:e0:23:35:d4:57:fb:a0:d5:d6:ae:3c:45:72:51:12:cd: + 36:bf:82:26:ba:fc:f5:06:67:53:fe:f5:96:a8:29:59:33:7f: + 27:8e:e6:3b:f9:5f:7c:c4:8f:8e:62:0d:13:e9:e0:5f:33:9f: + 7e:6d:2a:08:6d:be:80:dc:9a:38:13:9d:12:64:13:2d:d1:cf: + 21:7b:dc:9b:b4:fe:c0:4a:9c:ed:71:56:da:d8:c4:76:91:2e: + 6d:98:30:7a:bb:94:ff:e0:c1:45:7e:bf:3e:11:5b:6a:54:34: + 6c:a7:0c:78:52:f3:2c:00:f8:9a:6b:41:69:00:89:21:29:94: + 50:55:b1:d6:9c:82:4b:aa:60:06:d4:35:6c:dc:b7:3c:93:cd: + d8:f3:56:62:90:df:dc:33:59:d7:ab:eb:00:9e:1a:e1:2c:d7: + 2d:70:a8:38:dc:34:30:6c:59:71:2e:bf:85:cb:49:ef:95:73: + f9:d4:cc:ac:0a:ca:c1:c9:32:b5:68:10:4b:fe:5e:73:fd:7f: + c2:53:19:9f:8c:65:86:8f:84:70:83:cc:3d:f0:81:e6:15:b5: + ab:31:94:d9:68:d4:6a:44:d5:c7:31:78:82:86:a6:9d:f1:6e: + 29:b1:ca:12:85:d5:3a:8f:42:dd:d1:10:f6:e9:f8:53:27:75: + c4:7c:63:f1:6b:d0:72:4e:fd:22:7e:7f:4b:69:7d:85:da:a9: + e1:eb:9d:e0:16:98:07:ab:4d:44:ac:65:52:7b:52:0a:f0:8d: + ae:24:76:c2:b0:cd:b0:cb:90:e7:e2:89:45:e1:b9:d6:f3:34: + 3b:3c:ca:9a:54:9c:80:81:da:a9:b5:5e:34:56:48:f5:7e:93: + a6:c3:b6:6c:b1:19:5f:6c:72:a8:ac:38:c6:ba:de:38:81:19: + 46:82:6b:99:7d:d5:43:fe:2e:f6:18:f5:e1:06:ed:ef:6f:e4: + 85:8a:3a:cf:f1:5c:91:ce:73:54:3f:9d:5b:87:e3:3f:6c:ed: + 55:34:8f:cc:1a:88:2b:7f:87:cc:08:1e:c1:85:ba:70:9e:80: + 6a:7f:06:4a:5d:c8:7d:1a:a3:e9:69:4e:b9:b5:a3:f0:6a:f5: + 53:47:4e:13:b4:a4:05:7e:b0:fd:7e:b5:ba:ab:07:ac:ec:a0: + fc:4f:d9:d1:67:33:fd:5f:96:e6:26:71:6b:84:a9:9b:4c:78: + b1:fa:4d:cd:ff:db:36:68:d3:7b:83:36:31:7c:a7:fe:15:f1: + e4:c7:47:7f:f1:d9:76:13:ac:fe:5c:f0:29:41:8b:90:4c:1c: + cb:7f:1d:ea:16:2d:52:3b + +#### One-item revocation list + + GET /puppet-ca/v1/certificate_revocation_list/ca + + HTTP/1.1 200 OK + Content-Type: text/plain + + -----BEGIN X509 CRL----- + MIICnDCBhQIBATANBgkqhkiG9w0BAQUFADAfMR0wGwYDVQQDDBRQdXBwZXQgQ0E6 + IGxvY2FsaG9zdBcNMTMxMDA3MTk0ODQwWhcNMTgxMDA2MTk0ODQxWjAiMCACAQUX + DTEzMTAwNzE5NDg0MVowDDAKBgNVHRUEAwoBAaAOMAwwCgYDVR0UBAMCAQEwDQYJ + KoZIhvcNAQEFBQADggIBALrh49WNdmrJOPCRntD1nxCObmqZgl8ZwTv7TO9VkmCG + Ksvo8zR2aTIOH9VUKqWrE0squhtFJXl8dxL4PR1RiLbmhO7dp+NHdu8ejTQpoOTp + h69xbQFT3oHcIdn2cBGrLJQcZgXsiswT0KJ8nuw6eDO93yXDrguSUdou99M99wTw + 2nn1kUQKW9b0vUI7t2ADF5U8/DES+1IrvBq2IEHmg4+ekZRCxeJMuqd1R13gymcJ + osSPbRgIjCli6zD3aK4Nq5OMMpVLV/VVPwyQb4GwW4Wj5iyNAp8d/EAqtZ21ZHUi + nvuXmRtUWHJwfi40D5T2GQXxuUjB4pnh8cFq7f89iUvqoCwFo7nRIacrrweNFMYD + GxVJVMfz4PkP66ckIPQ5Uuey92dg5p2w4b2cp8NstxMdgcc3KAF483ItKA8uIDuU + 1dbzw1v2k5qUjoImueHwKolbLmPyYmvFp7hbnV+WpFbvGjyIfW3BMankDEv4ig0L + MCw6n2GKv1hSWM6Mrk8Ja1yYOFLsjI0RoVCZsf1iNiRT28haldXVTPyNtct9mGAv + 6az5W/nyixIPrrHubTx28zhmuHZx6y3hQMCLmuYOT+e7F/eFsYXVEjuJjxjr33uA + O/ii4EkTls1gzvonOtoBoGElzQAogrZI3HXCwFYvU2whLKr9cwv5bpRkUfPCMQ4n + -----END X509 CRL----- + + > openssl crl -inform PEM -in 1revoked.crl -text -noout + Certificate Revocation List (CRL): + Version 2 (0x1) + Signature Algorithm: sha1WithRSAEncryption + Issuer: /CN=Puppet CA: localhost + Last Update: Oct 7 19:48:40 2013 GMT + Next Update: Oct 6 19:48:41 2018 GMT + CRL extensions: + X509v3 CRL Number: + 1 + Revoked Certificates: + Serial Number: 05 + Revocation Date: Oct 7 19:48:41 2013 GMT + CRL entry extensions: + X509v3 CRL Reason Code: + Key Compromise + Signature Algorithm: sha1WithRSAEncryption + ba:e1:e3:d5:8d:76:6a:c9:38:f0:91:9e:d0:f5:9f:10:8e:6e: + 6a:99:82:5f:19:c1:3b:fb:4c:ef:55:92:60:86:2a:cb:e8:f3: + 34:76:69:32:0e:1f:d5:54:2a:a5:ab:13:4b:2a:ba:1b:45:25: + 79:7c:77:12:f8:3d:1d:51:88:b6:e6:84:ee:dd:a7:e3:47:76: + ef:1e:8d:34:29:a0:e4:e9:87:af:71:6d:01:53:de:81:dc:21: + d9:f6:70:11:ab:2c:94:1c:66:05:ec:8a:cc:13:d0:a2:7c:9e: + ec:3a:78:33:bd:df:25:c3:ae:0b:92:51:da:2e:f7:d3:3d:f7: + 04:f0:da:79:f5:91:44:0a:5b:d6:f4:bd:42:3b:b7:60:03:17: + 95:3c:fc:31:12:fb:52:2b:bc:1a:b6:20:41:e6:83:8f:9e:91: + 94:42:c5:e2:4c:ba:a7:75:47:5d:e0:ca:67:09:a2:c4:8f:6d: + 18:08:8c:29:62:eb:30:f7:68:ae:0d:ab:93:8c:32:95:4b:57: + f5:55:3f:0c:90:6f:81:b0:5b:85:a3:e6:2c:8d:02:9f:1d:fc: + 40:2a:b5:9d:b5:64:75:22:9e:fb:97:99:1b:54:58:72:70:7e: + 2e:34:0f:94:f6:19:05:f1:b9:48:c1:e2:99:e1:f1:c1:6a:ed: + ff:3d:89:4b:ea:a0:2c:05:a3:b9:d1:21:a7:2b:af:07:8d:14: + c6:03:1b:15:49:54:c7:f3:e0:f9:0f:eb:a7:24:20:f4:39:52: + e7:b2:f7:67:60:e6:9d:b0:e1:bd:9c:a7:c3:6c:b7:13:1d:81: + c7:37:28:01:78:f3:72:2d:28:0f:2e:20:3b:94:d5:d6:f3:c3: + 5b:f6:93:9a:94:8e:82:26:b9:e1:f0:2a:89:5b:2e:63:f2:62: + 6b:c5:a7:b8:5b:9d:5f:96:a4:56:ef:1a:3c:88:7d:6d:c1:31: + a9:e4:0c:4b:f8:8a:0d:0b:30:2c:3a:9f:61:8a:bf:58:52:58: + ce:8c:ae:4f:09:6b:5c:98:38:52:ec:8c:8d:11:a1:50:99:b1: + fd:62:36:24:53:db:c8:5a:95:d5:d5:4c:fc:8d:b5:cb:7d:98: + 60:2f:e9:ac:f9:5b:f9:f2:8b:12:0f:ae:b1:ee:6d:3c:76:f3: + 38:66:b8:76:71:eb:2d:e1:40:c0:8b:9a:e6:0e:4f:e7:bb:17: + f7:85:b1:85:d5:12:3b:89:8f:18:eb:df:7b:80:3b:f8:a2:e0: + 49:13:96:cd:60:ce:fa:27:3a:da:01:a0:61:25:cd:00:28:82: + b6:48:dc:75:c2:c0:56:2f:53:6c:21:2c:aa:fd:73:0b:f9:6e: + 94:64:51:f3:c2:31:0e:27 + +#### No node name given + + GET /puppet-ca/v1/certificate_revocation_list + + HTTP/1.1 400 Bad Request + Content-Type: text/plain + + No request key specified in /puppet-ca/v1/certificate_revocation_list + +## Schema + +A `certificate_revocation_list` response body is not structured data according to any +standard scheme such as json/pson/yaml, so no schema is applicable. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_sign.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate_sign.md new file mode 100644 index 000000000..27a5fa887 --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_sign.md @@ -0,0 +1,138 @@ +--- +layout: default +title: "OpenVox Server CA API: Bulk Certificate Sign" +--- + +## Bulk Certificate Sign + +The `sign` endpoints allow an administrator to sign one or more pending certificate signing requests (CSRs) +in a single API call. These endpoints are alternatives to using the `puppetserver ca sign` CLI command. + +Both endpoints require a client certificate with the `pp_cli_auth: "true"` extension. + +## Sign specific certificates + +Sign one or more pending CSRs by certname. + +```text +POST /puppet-ca/v1/sign +Content-Type: application/json +``` + +The request body must contain a `certnames` array listing the certnames to sign. + +### Supported HTTP Methods + +POST + +### Supported Response Formats + +`application/json` + +### Parameters + +None (certnames are provided in the request body) + +### Access + +Requires a client certificate with the `pp_cli_auth: "true"` certificate extension. Access is controlled by +OpenVox Server's `ca.conf` authorization configuration. + +### Example + +#### Request + +```text +POST /puppet-ca/v1/sign +Content-Type: application/json + +{"certnames": ["one.example.com", "nocsrone.example.com"]} +``` + +#### Response + +```text +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "signed": ["one.example.com"], + "no-csr": ["nocsrone.example.com"], + "signing-errors": [] +} +``` + +The response always contains three arrays: + +- `signed` — certnames whose CSRs were successfully signed. +- `no-csr` — certnames for which no pending CSR was found. +- `signing-errors` — certnames whose CSRs failed one or more validation checks. Checks include + subject alternative name allowance, authorization extension allowance, unrecognized extensions, + and signature validity. Example: + +```text +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "signed": [], + "no-csr": [], + "signing-errors": ["badextension.example.com", "invalidsignature.example.com"] +} +``` + +A `200 OK` is returned even if some certnames appear in `no-csr` or `signing-errors`. A `422 Unprocessable +Entity` is returned if the request body does not conform to the expected schema (for example, if `certnames` +contains non-string values): + +```text +HTTP/1.1 422 Unprocessable Entity +Content-Type: application/json + +{"kind":"schema-violation","submitted":[42],"error":"[(not (instance? java.lang.String 42))]"} +``` + +## Sign all pending certificates + +Sign all currently pending CSRs at once. The server automatically discovers all pending CSRs in the +configured CSR directory — no request body is needed. + +### Supported HTTP Methods + +POST + +### Supported Response Formats + +`application/json` + +### Parameters + +None + +### Access + +Requires a client certificate with the `pp_cli_auth: "true"` certificate extension. Access is controlled by +OpenVox Server's `ca.conf` authorization configuration. + +### Example + +#### Request + +```text +POST /puppet-ca/v1/sign/all +``` + +#### Response + +```text +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "signed": ["one.example.com", "two.example.com"], + "no-csr": [], + "signing-errors": [] +} +``` + +If there are no pending CSRs, the `signed` array is empty and the response is still `200 OK`. diff --git a/docs/_openvox-server_9x/ca-api/v1/http_certificate_status.md b/docs/_openvox-server_9x/ca-api/v1/http_certificate_status.md new file mode 100644 index 000000000..1954d5fb5 --- /dev/null +++ b/docs/_openvox-server_9x/ca-api/v1/http_certificate_status.md @@ -0,0 +1,166 @@ +--- +layout: default +title: 'OpenVox Server CA API: Certificate Status' +--- + +# Certificate Status + +The `certificate status` endpoint allows a client to read or alter the +status of a certificate or pending certificate request. It is only +useful on the CA. + +## Find + + GET /puppet-ca/v1/certificate_status/:certname + Accept: application/json, text/pson + +Retrieve information about the specified certificate. Similar to `puppetserver ca list --certname `. + +## Search + + GET /puppet-ca/v1/certificate_statuses/:any_key?state=:state + Accept: application/json, text/pson + +Retrieve information about all known certificates. Similar to `puppetserver ca list --all`. A key is required but is ignored. + +### Parameters + +* `state` (optional): The certificate state by which to filter search results. Valid states are 'requested', 'signed', and 'revoked'. + +## Save + + PUT /puppet-ca/v1/certificate_status/:certname + Content-Type: text/pson + +Change the status of the specified certificate. The desired state +is sent in the body of the PUT request as a one-item PSON hash; the two +allowed complete hashes are: + +* `{"desired_state":"signed"}` (for signing a certificate signing request, +similar to `puppetserver ca sign`). To set the validity period of the signed +certificate, specify the `cert_ttl` key in the body of the request, with an +integer value. By default, this key specifies the number of seconds, but you can +specify another time unit. See +[configuration](/openvox/latest/configuration.html#configuration-settings) for a +list of Puppet's accepted time unit markers. +* `{"desired_state":"revoked"}` (for revoking a certificate, similar to +`puppetserver ca revoke`). + +Note that revoking a certificate does not clean up other info about the +host --- see the DELETE request for more information. + +## Delete + + DELETE /puppet-ca/v1/certificate_status/:hostname + Accept: application/json, text/pson + +Cause the certificate authority to discard all SSL information regarding +a host (including any certificates, certificate requests, and keys). +This does not revoke the certificate if one is present; if you wish to +emulate the behavior of `puppet cert --clean`, you must PUT a +`desired_state` of `revoked` before deleting the host’s SSL information. + +If the deletion was successful, it returns a string listing the deleted +classes like + + "Deleted for myhost: Puppet::SSL::Certificate, Puppet::SSL::Key" + +Otherwise it returns + + "Nothing was deleted" + +### Supported HTTP Methods + +This endpoint is disabled in the default configuration. It is +recommended to be careful with this endpoint, as it can allow control +over the certificates used by the puppet master. + +GET, PUT, DELETE + + +### Supported Response Formats + +`application/json`, `text/pson`, `pson` + +This endpoint can produce yaml as well, but the returned data is +incomplete. + +### Examples + +#### Certificate information + + GET /puppet-ca/v1/certificate_status/mycertname + + HTTP/1.1 200 OK + Content-Type: text/pson + + { + "name":"mycertname", + "state":"signed", + "fingerprint":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "fingerprints":{ + "default":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "SHA1":"77:E6:5A:7E:DD:83:78:DC:F8:51:E3:8B:12:71:F4:57:F1:C2:34:AE", + "SHA256":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "SHA512":"CA:A0:8C:B9:FE:9D:C2:72:18:57:08:E9:4B:11:B7:BC:4E:F7:52:C8:9C:76:03:45:B4:B6:C5:D2:DC:E8:79:43:D7:71:1F:5C:97:FA:B2:F3:ED:AE:19:BD:A9:3B:DB:9F:A5:B4:8D:57:3F:40:34:29:50:AA:AA:0A:93:D8:D7:54" + }, + "dns_alt_names":["DNS:puppet","DNS:mycertname"] + } + +#### Search unsigned certs (CSRs) + + GET /puppet-ca/v1/certificate_statuses/ignored?state=requested + + HTTP/1.1 200 OK + Content-Type: text/pson + + [ + { + "name":"mycertname1", + "state":"requested", + "fingerprint":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "fingerprints":{ + "default":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "SHA1":"77:E6:5A:7E:DD:83:78:DC:F8:51:E3:8B:12:71:F4:57:F1:C2:34:AE", + "SHA256":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "SHA512":"CA:A0:8C:B9:FE:9D:C2:72:18:57:08:E9:4B:11:B7:BC:4E:F7:52:C8:9C:76:03:45:B4:B6:C5:D2:DC:E8:79:43:D7:71:1F:5C:97:FA:B2:F3:ED:AE:19:BD:A9:3B:DB:9F:A5:B4:8D:57:3F:40:34:29:50:AA:AA:0A:93:D8:D7:54" + }, + "dns_alt_names":[] + }, + { + "name":"mycertname2", + "state":"requested", + "fingerprint":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "fingerprints":{ + "default":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "SHA1":"77:E6:5A:7E:DD:83:78:DC:F8:51:E3:8B:12:71:F4:57:F1:C2:34:AE", + "SHA256":"A6:44:08:A6:38:62:88:5B:32:97:20:49:8A:4A:4A:AD:65:C3:3E:A2:4C:30:72:73:02:C5:F3:D4:0E:B7:FC:2F", + "SHA512":"CA:A0:8C:B9:FE:9D:C2:72:18:57:08:E9:4B:11:B7:BC:4E:F7:52:C8:9C:76:03:45:B4:B6:C5:D2:DC:E8:79:43:D7:71:1F:5C:97:FA:B2:F3:ED:AE:19:BD:A9:3B:DB:9F:A5:B4:8D:57:3F:40:34:29:50:AA:AA:0A:93:D8:D7:54" + }, + "dns_alt_names":[] + } + ] + +#### Revoking a certificate + + PUT /puppet-ca/v1/certificate_status/mycertname HTTP/1.1 + Content-Type: text/pson + Content-Length: 27 + + {"desired_state":"revoked"} + +This has no meaningful return value. + + +#### Deleting the certificate information + + DELETE /puppet-ca/v1/certificate_status/mycertname HTTP/1.1 + +Gets the response: + + "Deleted for mycertname: Puppet::SSL::Certificate, Puppet::SSL::Key" + +## Schema + +Find and search operations return objects which +conform to [the host schema.](/openvox/latest/schemas/host.json) diff --git a/docs/_openvox-server_9x/ca_basic_foss.png b/docs/_openvox-server_9x/ca_basic_foss.png new file mode 100644 index 000000000..a611b8982 Binary files /dev/null and b/docs/_openvox-server_9x/ca_basic_foss.png differ diff --git a/docs/_openvox-server_9x/ca_external_root_foss.png b/docs/_openvox-server_9x/ca_external_root_foss.png new file mode 100644 index 000000000..9604f13f8 Binary files /dev/null and b/docs/_openvox-server_9x/ca_external_root_foss.png differ diff --git a/docs/_openvox-server_9x/certificate_renewal.markdown b/docs/_openvox-server_9x/certificate_renewal.markdown new file mode 100644 index 000000000..3179584c0 --- /dev/null +++ b/docs/_openvox-server_9x/certificate_renewal.markdown @@ -0,0 +1,216 @@ +--- +layout: default +title: "Renewing and regenerating certificates" +--- + +Every certificate in an OpenVox deployment has a fixed lifetime. By default, certificates that the running CA signs for agents and compilers are valid for 5 years, while the CA certificate itself is valid for 15 years. When a certificate expires, TLS connections start failing with errors such as `certificate verify failed` or `certificate has expired`, and agent runs stop working. + +This page explains how to find out which certificate expired, how to replace an expired host or agent certificate, how to turn on automatic renewal to prevent future expirations, and how to extend an expired CA certificate without reissuing every agent certificate. + +> **Note:** Both lifetimes are controlled by the same setting, whose name is misleading. The [`ca_ttl`](/openvox/latest/configuration.html#ca_ttl) setting in `puppet.conf` (default `5y`) controls the lifetime of certificates the CA *signs*, not the CA certificate itself. +> The CA certificate's 15-year lifetime is a built-in fallback that `puppetserver ca setup` uses when `ca_ttl` is not set in `puppet.conf`; if you set `ca_ttl` before running setup, the CA certificate uses that value instead. +> The same fallback applies to any certificate the `puppetserver ca` CLI creates while the server is stopped, including the primary server's own certificate, so it can outlive the 5-year agent certificates. + +## Find out which certificate expired + +Check the expiration dates with `openssl`. On the primary server: + +```console +openssl x509 -enddate -noout -in "$(puppet config print hostcert)" +openssl x509 -enddate -noout -in /etc/puppetlabs/puppetserver/ca/ca_crt.pem +``` + +On an agent, check its own certificate and its copy of the CA certificate: + +```console +openssl x509 -enddate -noout -in "$(puppet config print hostcert)" +openssl x509 -enddate -noout -in "$(puppet config print localcacert)" +``` + +To inspect a certificate in full (subject, issuer, alt names, and validity), use `openssl x509 -in .pem -text -noout`. If `ca_crt.pem` contains a bundle of several certificates (the default layout uses an intermediate signing cert plus a root cert), `openssl x509` only shows the first one; use `openssl storeutl -noout -text ca_crt.pem` to print them all. + +With default settings, a certificate that expired 5 years after it was issued is a host certificate, and the CA certificate is still valid: the CA is not due to expire until 15 years after it was created. + +## Regenerate the primary server's certificate + +Use this procedure when the expired certificate is the primary server's own host certificate and the server is also the CA. The new certificate is signed by the existing CA, so agents are unaffected and nothing needs to change on other nodes. + +1. Stop OpenVox Server and back up the SSL directory: + + ```console + systemctl stop puppetserver + cp -a /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppet/ssl.bak + ``` + +1. Remove the expired certificate and its keys, and the CA's copy of the signed certificate: + + ```console + CERTNAME="$(puppet config print certname)" + rm /etc/puppetlabs/puppet/ssl/certs/"$CERTNAME".pem + rm /etc/puppetlabs/puppet/ssl/private_keys/"$CERTNAME".pem + rm /etc/puppetlabs/puppet/ssl/public_keys/"$CERTNAME".pem + rm /etc/puppetlabs/puppetserver/ca/signed/"$CERTNAME".pem + ``` + +1. Generate a new certificate. The `--ca-client` flag makes this work offline, signing directly with the CA's key while the server is stopped: + + ```console + puppetserver ca generate --certname "$CERTNAME" --ca-client + ``` + + The original certificate usually contains subject alternative names, and `puppetserver ca generate` does not carry them over. Check the backed-up certificate with `openssl x509 -text -noout` and pass every name agents use, such as a load balancer name, a CNAME, or a `puppet` DNS alias, with `--subject-alt-names`. + A certificate created at setup time includes `DNS:puppet` by default, so regenerating without the flag drops that name and agents that connect to the server as `puppet` reject the new certificate. + Don't rely on `puppet config print dns_alt_names` here: the setting is usually empty even when the certificate has alternative names. + + If the command reports that it could not determine whether Puppet Server is online (for example, when the `server` hostname does not resolve), confirm that the service is stopped and rerun with `--force`. + +1. Start OpenVox Server and verify the new certificate: + + ```console + systemctl start puppetserver + openssl x509 -enddate -noout -in "$(puppet config print hostcert)" + ``` + + Then run `puppet agent -t` on an agent to confirm that agents can connect. + +If OpenVoxDB runs on the same host, it keeps its own copies of the certificate, key, and CA certificate in `/etc/puppetlabs/puppetdb/ssl/`. Refresh them and restart OpenVoxDB: + +```console +puppetdb ssl-setup -f +systemctl restart puppetdb +``` + +## Regenerate an agent certificate + +Use this procedure when an agent's certificate has expired, or when a compiler's certificate has expired and the CA runs on a different server. An expired client certificate can't authenticate, so the agent needs a new one signed by the CA. + +1. On the CA server, revoke and remove the old certificate: + + ```console + puppetserver ca clean --certname + ``` + +1. On the agent, delete the expired certificate and key: + + ```console + puppet ssl clean + ``` + +1. On the agent, request a new certificate: + + ```console + puppet ssl bootstrap + ``` + + This submits a new certificate signing request, then waits and retries every 2 minutes until the certificate is signed. Leave it running while you sign the request in the next step; it downloads the certificate on its next retry. + To submit the request without waiting instead, run `puppet ssl bootstrap --waitforcert 0`; it submits the request and then exits with an error noting the certificate has not been signed yet, which is expected. Sign the request, then run the command again to download the certificate. A regular `puppet agent -t` run submits the same request. + +1. If you don't use autosigning, sign the request from a session on the CA server: + + ```console + puppetserver ca sign --certname + ``` + +If the certificate needs subject alternative names, the CA must have `allow-subject-alt-names: true` in the `certificate-authority` section of [`ca.conf`](config_file_ca.html), and the agent must request them, for example with `puppet ssl bootstrap --dns_alt_names ,`. + +For a compiler, stop OpenVox Server on the compiler before cleaning its certificate and start it again after the new certificate is in place. If OpenVoxDB shares the host, refresh its certificate copies as shown in the previous section. + +## Turn on automatic renewal + +OpenVox Server supports automatic certificate renewal, which prevents host certificates from ever reaching their expiration date. It is off by default. +The packaged [`ca.conf`](config_file_ca.html) already contains the settings in its `certificate-authority` section; set `allow-auto-renewal` to `true` and restart OpenVox Server: + +```text +certificate-authority: { + allow-auto-renewal: true + auto-renewal-cert-ttl: "60d" +} +``` + +With auto-renewal enabled, the CA issues certificates to renewal-capable agents with the shorter `auto-renewal-cert-ttl` lifetime instead of the `ca_ttl` value. +The packaged configuration sets `auto-renewal-cert-ttl` to 60 days; when the setting is absent, the built-in default is 90 days. +All OpenVox agents support renewal: when an agent's certificate is within `hostcert_renewal_interval` (30 days by default) of expiring, the agent requests a renewed certificate during its regular run and switches to it transparently, using the [certificate renewal endpoint](ca-api/v1/http_certificate_renewal.html). + +You can also renew a certificate on demand by running `puppet ssl renew_cert` on the agent. It uses the same renewal endpoint and works on any still-valid certificate once renewal is enabled, including certificates issued before you enabled it, and replaces the certificate with one that has the `auto-renewal-cert-ttl` lifetime. +Pass `--if-expiring-in ` (for example `30d`) to renew only when the certificate is close to expiry. +If `allow-auto-renewal` is not enabled on the CA, the command does nothing and still exits successfully, so check the certificate's expiration date afterward to confirm the renewal happened. + +Renewal authenticates with the agent's current certificate, so it only works while that certificate is still valid. It does not work for a certificate that has already expired: use the regeneration procedures above first, then enable auto-renewal. + +## Extend an expired CA certificate + +An expired CA certificate does not require rebuilding the CA and reissuing every certificate in the deployment. Because every host certificate was signed by the CA's private key, re-signing the CA certificate with the same key and subject gives it a new validity period while existing host certificates remain valid. +Only the CA certificate file changes, and you then distribute it to the rest of the deployment. + +The [`puppetlabs/ca_extend`](https://forge.puppet.com/modules/puppetlabs/ca_extend) module automates this procedure and works with open source deployments. +Its `extend_ca_cert` plan re-signs the CA certificate on the primary server (pass `regen_primary_cert=true` if the server's own host certificate has also expired) and repairs an expired CRL along the way, and its `upload_ca_cert` plan distributes the refreshed certificate to agents. +If you can run OpenBolt against your infrastructure, use the module. The manual procedure below performs the same steps. + +> **Warning:** A mistake here can break authentication for the whole deployment. Back up the CA directory (`/etc/puppetlabs/puppetserver/ca`) before you start, and test the procedure in a test environment first if you can. +> If your CA uses the default intermediate layout, `ca_crt.pem` is a bundle of the signing certificate and the root certificate. The same re-signing technique applies to each certificate in the bundle, but you must re-sign the root with the root's key (`root_key.pem`) and the intermediate with the root as issuer, then rebuild the bundle in the same order. + +To extend a CA with a single self-signed certificate manually, on the CA server: + +1. Confirm the CA key matches the CA certificate; the two digests must be identical: + + ```console + cd /etc/puppetlabs/puppetserver/ca + openssl rsa -noout -modulus -in ca_key.pem | openssl md5 + openssl x509 -noout -modulus -in ca_crt.pem | openssl md5 + ``` + +1. Back up the certificate, then turn it into a new CSR signed by the same key: + + ```console + cp -p ca_crt.pem ca_crt.pem.bak + openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem -out ca_csr.pem + ``` + +1. Re-sign it with the CA extensions and a new validity period (15 years here): + + ```console + cat > extension.cnf < **Note:** You can also use the [`puppetlabs-puppet_authorization`](https://forge.puppet.com/puppetlabs/puppet_authorization) module to manage the new `auth.conf` file's authorization rules in the new HOCON +> format, and the [`puppetlabs-hocon`](https://forge.puppet.com/puppetlabs/hocon) module to use OpenVox to manage HOCON-formatted settings in general. + +To configure how OpenVox Server authenticates requests, use the supported HOCON `auth.conf` file and authorization methods, and see the parameters and rule definitions in the +[HOCON Parameters](#hocon-parameters) section. + +You can find the OpenVox Server auth.conf file [here](https://github.com/openvoxproject/openvox-server/blob/master/ezbake/config/conf.d/auth.conf). + +> **Note:** Starting with OpenVox Server 9.0, the default `auth.conf` splits the +> `/puppet/v3/file_bucket_file` rule by HTTP method. Agents keep `head` and `put`, which +> is all they need to back files up. Reading content back out with `get` or `post` now +> requires a client certificate with the `pp_cli_auth: "true"` extension, the same +> extension that gates the CA administrative endpoints. If you restore or diff filebucket +> content remotely with an ordinary agent certificate, add a rule for that certname rather +> than widening the shipped rule. + +## HOCON example + +Here is an example authorization section using the HOCON configuration format: + +```hocon +authorization: { + version: 1 + rules: [ + { + match-request: { + path: "^/my_path/([^/]+)$" + type: regex + method: get + } + allow: [ node1, node2, node3, {extensions:{ext_shortname1: value1, ext_shortname2: value2}} ] + sort-order: 1 + name: "user-specific my_path" + }, + { + match-request: { + path: "/my_other_path" + type: path + } + allow-unauthenticated: true + sort-order: 2 + name: "my_other_path" + }, + ] +} +``` + +For a more detailed example of how to use the HOCON configuration format, see +[Configuring The Authorization Service](https://github.com/openvoxproject/trapperkeeper-authorization/blob/master/doc/authorization-config.md). + +For descriptions of each setting, see the following sections. + +## HOCON parameters + +Use the following parameters when writing or migrating custom authorization rules using the new HOCON format. + +### `version` + +The `version` parameter is required. In this initial release, the only supported value is `1`. + +### `allow-header-cert-info` + +This optional `authorization` section parameter determines whether to enable [external SSL termination](./external_ssl_termination.html) on all HTTP endpoints that OpenVox Server handles, including those served +by the "server" service, the certificate authority API, and the Puppet Admin API. It also controls how OpenVox Server derives the user's identity for authorization purposes. The default value is `false`. + +If this setting is `true`, OpenVox Server ignores any presented certificate and relies completely on header data to authorize requests. + +> **Warning!** This is very insecure; **do not enable this parameter** unless you've secured your network to prevent **any** untrusted access to OpenVox Server. + +You cannot rename any of the `X-Client` headers when this setting is enabled, and you must specify identity through the `X-Client-Verify`, `X-Client-DN`, and `X-Client-Cert` headers. + +For more information, see [External SSL Termination](./external_ssl_termination.html#disable-https-for-puppet-server) in the OpenVox Server documentation and +[Configuring the Authorization Service](https://github.com/openvoxproject/trapperkeeper-authorization/blob/master/doc/authorization-config.md#allow-header-cert-info) in the `trapperkeeper-authorization` +documentation. + +### `rules` + +The required `rules` array of an OpenVox Server's HOCON `auth.conf` file determines how OpenVox Server responds to a request. +Each element is a map of settings pertaining to a rule, and when OpenVox Server receives a request, it evaluates that request against each rule looking for a match. + +You define each rule by adding parameters to the rule's [`match-request`](#match-request) section. A `rules` array can contain as many rules as you need, each with a single `match-request` section. + +If a request matches a rule in a `match-request` section, Puppet Server determines whether to allow or deny the request using the `rules` parameters that follow the rule's `match-request` section: + +- At least one of: + - [`allow`](#allow-allow-unauthenticated-and-deny) + - [`allow-unauthenticated`](#allow-allow-unauthenticated-and-deny) + - [`deny`](#allow-allow-unauthenticated-and-deny) +- [`sort-order`](#sort-order) (required) +- [`name`](#name) (required) + +If no rule matches, OpenVox Server denies the request by default and returns an HTTP 403/Forbidden response. + +#### `match-request` + +A `match-request` can take the following parameters, some of which are required: + +- **`path` and `type` (required):** A `match-request` rule must have a `path` parameter, which returns a match when a request's endpoint URL starts with or contains the `path` parameter's value. + The parameter can be a literal string or regular expression as defined in the required `type` parameter. + + ```hocon + # Regular expression to match a path in a URL. + path: "^/puppet/v3/report/([^/]+)$" + type: regex + + # Literal string to match the start of a URL's path. + path: "/puppet/v3/report/" + type: path + ``` + + > **Note:** While the HOCON format doesn't require you to wrap all string values with double quotation marks, some special characters commonly used in regular expressions --- such as `*` --- break HOCON + > parsing unless the entire value is enclosed in double quotes. + +- **`method`:** If a rule contains the optional `method` parameter, OpenVox Server applies that rule only to requests that use its value's listed HTTP methods. This parameter's valid values are `get`, `post`, + `put`, `delete`, and `head`, provided either as a single value or array of values. + + ```hocon + # Use GET and POST. + method: [get, post] + + # Use PUT. + method: put + ``` + + > **Note:** While the new HOCON format does not provide a direct equivalent to the [deprecated][] `method` parameter's `search` indirector, you can create the equivalent rule by passing GET and POST to + > `method` and specifying endpoint paths using the `path` parameter. + +- **`query-params`:** Use the optional query-params setting to provide the list of query parameters. Each entry is a hash of the param name followed by a list of its values. + +For example, this rule would match a request URL containing the `environment=production` or `environment=test` query parameters: + +```hocon +query-params: { + environment: [ production, test ] +} +``` + +#### `allow`, `allow-unauthenticated`, and `deny` + +After each rule's `match-request` section, it must also have an `allow`, `allow-unauthenticated`, or `deny` parameter. +(You can set both `allow` and `deny` parameters for a rule, though OpenVox Server always prioritizes `deny` over `allow` when a request matches both.) + +If a request matches the rule, OpenVox Server checks the request's authenticated "name" (see [`allow-header-cert-info`](#allow-header-cert-info)) against these parameters to determine what to do with the +request. + +- **`allow-unauthenticated`**: If this Boolean parameter is set to `true`, OpenVox Server allows the request --- even if it can't determine an authenticated name. **This is a potentially insecure configuration** + --- be careful when enabling it. A rule with this parameter set to `true` can't also contain the `allow` or `deny` parameters. +- **`allow`**: This parameter can take a single string value, an array of string values, a single map value with either an `extensions` or `certname` key, or an array of string and map values. + + The string values can contain: + - An exact domain name, such as `www.example.com`. + - A glob of names containing a `*` in the first segment, such as `*.example.com` or simply `*`. + - A regular expression surrounded by `/` characters, such as `/example/`. + - A backreference to a regular expression's capture group in the `path` value, if the rule also contains a `type` value of `regex`. For example, if the path for the rule were `"^/example/([^/]+)$"`, you can + make a backreference to the first capture group using a value like `$1.domain.org`. + + The map values can contain: + + - An `extensions` key that specifies an array of matching X.509 extensions. OpenVox Server authenticates the request only if each key in the map appears in the request, and each key's value exactly matches. + - A `certname` key equivalent to a bare string. + + If the request's authenticated name matches the parameter's value, OpenVox Server allows it. + +> **Note:** If you are using OpenVox Server with the CA disabled, you must use OID values for the extensions. OpenVox Server will not be able to resolve [short names][] in this mode. + +- **`deny`**: This parameter can take the same types of values as the `allow` parameter, but refuses the request if the authenticated name matches --- even if the rule contains an `allow` value that also + matches. + +> Also, in the HOCON OpenVox Server authentication method, there is no directly equivalent behavior to the [deprecated][] `auth` parameter's `on` value. + +#### `sort-order` + +After each rule's `match-request` section, the required `sort-order` parameter sets the order in which OpenVox Server evaluates the rule by prioritizing it on a numeric value between 1 and 399 (to be evaluated +before default OpenVox rules) or 601 to 998 (to be evaluated after OpenVox), with lower-numbered values evaluated first. OpenVox Server secondarily sorts rules lexicographically by the `name` string value's +Unicode code points. + +```hocon +sort-order: 1 +``` + +#### `name` + +After each rule's `match-request` section, this required parameter's unique string value identifies the rule to OpenVox Server. The `name` value is also written to server logs and error responses returned to +unauthorized clients. + +```hocon +name: "my path" +``` + +> **Note:** If multiple rules have the same `name` value, OpenVox Server will fail to launch. diff --git a/docs/_openvox-server_9x/config_file_ca.markdown b/docs/_openvox-server_9x/config_file_ca.markdown new file mode 100644 index 000000000..d1573d44a --- /dev/null +++ b/docs/_openvox-server_9x/config_file_ca.markdown @@ -0,0 +1,61 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: ca.conf" +--- + +[`trapperkeeper-authorization`]: https://github.com/openvoxproject/trapperkeeper-authorization +[new `auth.conf`]: ./config_file_auth.html +[deprecated]: ./deprecated_features.html + +The `ca.conf` file configures settings for the OpenVox Server Certificate Authority (CA) service. For an overview, see [OpenVox Server Configuration](./configuration.html). + +> **Deprecation Note:** The `authorization-required` and `client-whitelist` settings are [deprecated][] in favor of authorization that is configured in the [new `auth.conf`][] file. + +## Signing settings + +The `allow-subject-alt-names` setting in the `certificate-authority` section enables you to sign certs with subject alternative names. It is false by default for security reasons, but can be enabled if you need +to sign certs with subject alternative names. `puppet cert sign` used to allow this via a flag, but `puppetserver ca sign` requires it to be configured in the config file. + +The `allow-authorization-extensions` setting in the `certificate-authority` section enables you to sign certs with authorization extensions. It is false by default for security reasons, but can be enabled if +you know you need to sign certs this way. `puppet cert sign` used to allow this via a flag, but `puppetserver ca sign` requires it to be configued in the config file. + +## Auto-renewal settings + +The `allow-auto-renewal` setting in the `certificate-authority` section enables automatic renewal of agent certificates. It is false by default. +When enabled, the CA issues certificates to renewal-capable agents with the lifetime from the `auto-renewal-cert-ttl` setting instead of the `ca_ttl` value, and agents renew their certificates automatically before they expire. +The packaged `ca.conf` sets `auto-renewal-cert-ttl` to 60 days; when the setting is absent, the built-in default is 90 days. See [Renewing and regenerating certificates](certificate_renewal.html) for details. + +The `certificate-authority` section also accepts a `ca-ttl` setting that controls the lifetime of newly signed certificates and takes precedence over the `ca_ttl` value in `puppet.conf`. + +## Infrastructure CRL settings + +OpenVox Server is able to create a separate CRL file containing only revocations of OpenVox infrastructure nodes. This behavior is turned off by default. To enable it, set `certificate-authority.enable-infra-crl` +to `true`. + +## Status settings (deprecated) + +The `certificate-status` setting in `ca.conf` provides [deprecated][] configuration options for access to the `certificate_status` and `certificate_statuses` HTTP endpoints. These endpoints allow certificates +to be signed, revoked, and deleted through HTTP requests, which provides full control over OpenVox's ability to securely authorize access. Therefore, you should **always** restrict access to `ca.conf`. + +The `certificate-status` setting takes two parameters: `authorization-required` and `client-whitelist`. If `authorization-required` is set to `true` or not set, **and** `client-whitelist` is set to an empty +list or not set, OpenVox Server uses the [authorization methods][`trapperkeeper-authorization`] and [new `auth.conf`][] format to control access to the administration API endpoints. + +- `authorization-required` determines whether a client certificate is required to access certificate status endpoints. If this parameter is set to `false`, all requests can access this API. If set to `true`, + only the clients whose certificate names are included in the `client-whitelist` setting can access the admin API. If this parameter is not specified but the `client-whitelist` parameter is, this parameter's + value defaults to `true`. +- `client-whitelist` contains a list of client certificate names that are whitelisted for access to the certificate status endpoints. OpenVox Server denies access to requests at these endpoints that do not + present a valid client certificate named in this list. + +## Example (Deprecated) + +If you are using the deprecated authorization methods, follow this structure to configure `certificate_status` and `certificate_statuses` endpoint access in `ca.conf`, whitelisting a client named `host1`: + +```text +certificate-authority: { + # deprecated in favor of auth.conf + certificate-status: { + authorization-required: true + client-whitelist: [host1] + } +} +``` diff --git a/docs/_openvox-server_9x/config_file_global.markdown b/docs/_openvox-server_9x/config_file_global.markdown new file mode 100644 index 000000000..dc7f2ee56 --- /dev/null +++ b/docs/_openvox-server_9x/config_file_global.markdown @@ -0,0 +1,17 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: global.conf" +--- + +The `global.conf` file contains global configuration settings for OpenVox Server. For an overview, see [OpenVox Server Configuration](./configuration.html). + +You shouldn't typically need to make changes to this file. However, you can change the `logging-config` path for the logback logging configuration file if necessary. For more information about the logback file, +see . + +## Example + +```text +global: { + logging-config: /etc/puppetlabs/puppetserver/logback.xml +} +``` diff --git a/docs/_openvox-server_9x/config_file_logbackxml.markdown b/docs/_openvox-server_9x/config_file_logbackxml.markdown new file mode 100644 index 000000000..236295baf --- /dev/null +++ b/docs/_openvox-server_9x/config_file_logbackxml.markdown @@ -0,0 +1,75 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: logback.xml" +--- + +OpenVox Server’s logging is routed through the Java Virtual Machine's [Logback library](http://logback.qos.ch/) and configured in an XML file typically named `logback.xml`. + +> **Note:** This document covers basic, commonly modified options for OpenVox Server logs. Logback is a powerful library with many options. For detailed information on configuring Logback, see the +> [Logback Configuration Manual](http://logback.qos.ch/manual/configuration.html). +> +> For advanced logging configuration tips specific to OpenVox Server, such as configuring Logstash or outputting logs in JSON format, see [Advanced Logging Configuration](./config_logging_advanced.html). + +## OpenVox Server logging + +By default, OpenVox Server logs messages and errors to `/var/log/puppetlabs/puppetserver/puppetserver.log`. The default log level is ‘INFO’, and OpenVox Server sends nothing to `syslog`. You can change OpenVox +Server's logging behavior by editing `/etc/puppetlabs/puppetserver/logback.xml`, and you can specify a different Logback config file in [`global.conf`](./config_file_global.html). + +You can restart the `puppetserver` service for changes to take effect, or enable [configuration scanning](#scan-and-scanperiod) to allow changes to be recognized at runtime. + +OpenVox Server also relies on Logback to manage, rotate, and archive Server log files. Logback archives Server logs when they exceed 10MB, and when the total size of all Server logs exceeds 1GB, it automatically +deletes the oldest logs. + +### Settings + +#### `level` + +To modify OpenVox Server's logging level, change the `level` attribute of the `root` element. By default, the logging level is set to `info`: + + + +Supported logging levels, in order from most to least information logged, are `trace`, `debug`, `info`, `warn`, and `error`. For instance, to enable debug logging for Puppet Server, change `info` to `debug`: + + + +OpenVox Server profiling data is included at the `debug` logging level. + +You can also change the logging level for JRuby logging from its defaults of `error` and `info` by setting the `level` attribute of the `jruby` element. For example, to enable debug logging for JRuby, set the +attribute to `debug`: + + + +#### Logging location + +You can change the file to which OpenVox Server writes its logs in the `appender` section named `F1`. By default, the location is set to `/var/log/puppetlabs/puppetserver/puppetserver.log`: + + ... + + /var/log/puppetlabs/puppetserver/puppetserver.log + ... + +To change this to `/var/log/puppetserver.log`, modify the contents of the `file` element: + + /var/log/puppetserver.log + +The user account that owns the OpenVox Server process must have write permissions to the destination path. + +#### `scan` and `scanPeriod` + +Logback supports noticing and reloading configuration changes without requiring a restart, a feature Logback calls **scanning**. To enable this, set the `scan` and `scanPeriod` attributes in the +`` element of `logback.xml`: + + + +Due to a bug in Logback, the `scanPeriod` must be set to a value; setting only `scan="true"` will not enable configuration scanning. Scanning is enabled by +default in the `logback.xml` configuration packaged with OpenVox Server. + +**Note:** The HTTP request log does not currently support the scan feature. Adding the `scan` or `scanPeriod` settings to `request-logging.xml` will have no effect. + +## HTTP request logging + +OpenVox Server logs HTTP traffic separately, and this logging is configured in a different Logback configuration file located at `/etc/puppetlabs/puppetserver/request-logging.xml`. To specify a different Logback +configuration file, change the `access-log-config` setting in OpenVox Server's [`webserver.conf`](./config_file_webserver.html) file. + +The HTTP request log uses the same Logback configuration format and settings as the OpenVox Server log. It also lets you configure what it logs using patterns, which follow Logback's +[`PatternLayout` format](http://logback.qos.ch/manual/layouts.html#AccessPatternLayout). diff --git a/docs/_openvox-server_9x/config_file_master.markdown b/docs/_openvox-server_9x/config_file_master.markdown new file mode 100644 index 000000000..72ae633d4 --- /dev/null +++ b/docs/_openvox-server_9x/config_file_master.markdown @@ -0,0 +1,62 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: master.conf" +--- + +[new `auth.conf`]: ./config_file_auth.html +[deprecated]: ./deprecated_features.html +[`puppetserver.conf`]: ./config_file_puppetserver.html + +The `master.conf` file configures how OpenVox Server handles [deprecated][] authorization methods for server endpoints. For an overview, see [OpenVox Server Configuration](./configuration.html). + +> **Deprecation Note:** This file contains only the `allow-header-cert-info` parameter, and is deprecated in favor of authorization settings that are configured in the [new +> `auth.conf`][] file. Because this setting is deprecated, a default `master.conf` file is no longer included in the OpenVox Server package. + +In `master.conf`, the `allow-header-cert-info` setting determines whether OpenVox Server should use authorization info from the `X-Client-Verify`, `X-Client-DN`, and `X-Client-Cert` HTTP headers. Its default +value is `false`. + +The `allow-header-cert-info` setting is used to enable [external SSL termination](./external_ssl_termination.html). If the setting's value is set to `true`, OpenVox Server will ignore any certificate presented +to the Jetty web server, and will rely on header data to authorize requests. This is very dangerous unless you've secured your network to prevent any untrusted access to OpenVox Server. + +When using the `allow-header-cert-info` setting in `master.conf`, you can change OpenVox's `ssl_client_verify_header` parameter to use another header name instead of `X-Client-Verify`. The `ssl_client_header` +parameter can rename `X-Client-DN`. The `X-Client-Cert` header can't be renamed. + +The `allow-header-cert-info` parameter in `master.conf` applies only to HTTP endpoints served by the "server" service. The applicable endpoints include those listed in +[OpenVox V3 HTTP API](./http_api_index.html#openvox-v3-http-api). It does not apply to the endpoints listed in +[CA V1 HTTP API](./http_api_index.html#ca-v1-http-api) or to any [OpenVox Admin API][`puppetserver.conf`] endpoints. + +## Supported Authorization Workflow + +If you instead enable the `auth.conf` authorization method, the value of the `allow-header-cert-info` parameter in `auth.conf` controls how the user's identity is derived for +authorization purposes. In this case, OpenVox Server ignores the value of the `allow-header-cert-info` parameter in `master.conf`. + +When using the `allow-header-cert-info` parameter in `auth.conf`, none of the `X-Client` headers can be renamed. Identity must be specified through the `X-Client-Verify`, `X-Client-DN`, and `X-Client-Cert` +headers. + +The `allow-header-cert-info` parameter in `auth.conf`, applies to all HTTP endpoints that OpenVox Server handles, including those served by the "server" service, the CA API, and the OpenVox Admin API. + +For additional information on the `allow-header-cert-info` parameter in `auth.conf`, see [OpenVox Server Configuration Files: `auth.conf`][new `auth.conf`] and +[Configuring the Authorization Service in the `trapperkeeper-authorization` documentation](https://github.com/openvoxproject/trapperkeeper-authorization/blob/master/doc/authorization-config.md#allow-header-cert-info). + +### HOCON `auth.conf` Example + +```hocon +authorization: { + version: 1 + # allow-header-cert-info: false + rules: [ + { + # Allow nodes to retrieve their own catalog + match-request: { + path: "^/puppet/v3/catalog/([^/]+)$" + type: regex + method: [get, post] + } + allow: "$1" + sort-order: 500 + name: "puppetlabs catalog" + }, + ... + ] +} +``` diff --git a/docs/_openvox-server_9x/config_file_metrics.markdown b/docs/_openvox-server_9x/config_file_metrics.markdown new file mode 100644 index 000000000..f760e2c16 --- /dev/null +++ b/docs/_openvox-server_9x/config_file_metrics.markdown @@ -0,0 +1,60 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: metrics.conf" +--- + +The `metrics.conf` file configures OpenVox Server's [metrics services](./puppet_server_metrics.html) and [v2 metrics API](./metrics-api/v2/metrics_api.html). + +## Settings + +All settings in the file are contained in a HOCON `metrics` section. + +- `server-id`: A unique identifier to be used as part of the namespace for metrics that this server produces. + +- `registries`: A section that contains settings to control which metrics are reported, and how they're reported. + - ``: A section named for a registry that contains its settings. In OpenVox Server's case, this section should be `puppetserver`. + - `metrics-allowed`: An array of metrics to report. See the [metrics documentation](./puppet_server_metrics.html) for details about individual metrics. + - `reporters`: Can contain `jmx` and `graphite` sections with a single Boolean `enabled` setting to enable or disable each reporter type. +- `reporters`: Configures reporters that distribute metrics to external services or viewers. + - `graphite`: Contains settings for the Graphite reporter. + - `host`: A string containing the Graphite server's hostname or IP address. + - `port`: Contains the Graphite service's port number. + - `update-interval-seconds`: Sets the interval on which OpenVox Server will send metrics to the Graphite server. + +## Example + +OpenVox Server ships with a default `metrics.conf` file in OpenVox Server's `conf.d` directory, similar to the below example with additional comments. + +```text +metrics: { + server-id: localhost + registries: { + puppetserver: { + # specify metrics to allow in addition to those in the default list + #metrics-allowed: ["compiler.compile.production"] + + reporters: { + jmx: { + enabled: true + } + # enable or disable Graphite metrics reporter + #graphite: { + # enabled: true + #} + } + + } + } + + reporters: { + #graphite: { + # # graphite host + # host: "127.0.0.1" + # # graphite metrics port + # port: 2003 + # # how often to send metrics to graphite + # update-interval-seconds: 5 + #} + } +} +``` diff --git a/docs/_openvox-server_9x/config_file_product.markdown b/docs/_openvox-server_9x/config_file_product.markdown new file mode 100644 index 000000000..4f81a3135 --- /dev/null +++ b/docs/_openvox-server_9x/config_file_product.markdown @@ -0,0 +1,32 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: product.conf" +--- + +The `product.conf` file contains settings that determine how OpenVox Server interacts with Puppet, Inc., such as automatic update checking and analytics data collection. + +## Settings + +The `product.conf` file doesn't exist in a default OpenVox Server installation; to configure its settings, you must create it in OpenVox Server's `conf.d` directory +(located by default at `/etc/puppetlabs/puppetserver/conf.d`). This file is a [HOCON-formatted](https://github.com/typesafehub/config/blob/master/HOCON.md) configuration file with the following settings: + +- Settings in the `product` section configure update checking and analytics data collection: + + - `check-for-updates`: If set to `false`, OpenVox Server will not automatically check for updates, and will not send analytics data to Puppet. + If this setting is unspecified (default) or set to `true`, OpenVox Server checks for updates upon start or restart, and every 24 hours thereafter, by sending the following data to Puppet: + - Product name + - OpenVox Server version + - IP address + - Data collection timestamp + OpenVox requests this data as one of the many ways we learn about and work with our community. The more we know about how you use OpenVox, the better we can address your needs. + No personally identifiable information is collected, and the data we collect is never used or shared outside of OpenVox. + +### Example + +``` hocon +# Disabling automatic update checks and corresponding analytic data collection + +product: { + check-for-updates: false +} +``` diff --git a/docs/_openvox-server_9x/config_file_puppetserver.markdown b/docs/_openvox-server_9x/config_file_puppetserver.markdown new file mode 100644 index 000000000..2d9a3f144 --- /dev/null +++ b/docs/_openvox-server_9x/config_file_puppetserver.markdown @@ -0,0 +1,186 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: puppetserver.conf" +--- + +[configuration directory]: /openvox/latest/dirs_confdir.html +[code directory]: /openvox/latest/dirs_codedir.html +[cache directory]: /openvox/latest/dirs_vardir.html +[`auth.conf` documentation]: ./config_file_auth.html +[static catalogs]: /openvox/latest/static_catalogs.html +[file resource]: /openvox/latest/type.html#file +[`static_file_content`]: ./puppet-api/v3/static_file_content.html + +The `puppetserver.conf` file contains settings for OpenVox Server software. For an overview, see [OpenVox Server Configuration](./configuration.html). + +## Settings + +> **Note:** Under most conditions, you won't change the default settings for `master-conf-dir` or `master-code-dir`. However, if you do, also change the equivalent OpenVox settings (`confdir` or `codedir`) to +> ensure that commands like `puppetserver ca` and `puppet module` use the same directories as OpenVox Server. You must also specify the non-default `confdir` when running commands, because that setting must be +> set before OpenVox tries to find its config file. + +- The `jruby-puppet` settings configure the interpreter. + + - `ruby-load-path`: The location where OpenVox Server expects to find OpenVox, OpenFact, and other components. + + - `gem-home`: The location where JRuby looks for gems. It is also used by the `puppetserver gem` command line tool. If nothing is specified, JRuby uses the OpenVox default + `/opt/puppetlabs/server/data/puppetserver/jruby-gems`. + + - `gem-path`: The complete "GEM_PATH" for jruby. If set, it should include the `gem-home` directory, as well as any other directories that gems can be loaded from (including the vendored gems directory for + gems that ship with OpenVox Server). The default value is + `["/opt/puppetlabs/server/data/puppetserver/jruby-gems", "/opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems", "/opt/puppetlabs/puppet/lib/ruby/vendor_gems"]`. + + - `environment-vars:` Optional. A map of environment variables which are made visible to Ruby code running within JRuby, for example, via the Ruby `ENV` class. + + By default, the only environment variables whose values are set into JRuby from the shell are `HOME` and `PATH`. + + The default value for the `GEM_HOME` environment variable in JRuby is set from the value provided for the `jruby-puppet.gem-home` key. + + Any variable set from the map for the `environment-vars` key overrides these defaults. Avoid overriding `HOME`, `PATH`, or `GEM_HOME` here because these values are already configurable via the shell or + `jruby-puppet.gem-home`. + + - `master-conf-dir`: Optional. The path to the OpenVox [configuration directory][]. The default is `/etc/puppetlabs/puppet`. + + - `master-code-dir`: Optional. The path to the OpenVox [code directory][]. The default is `/etc/puppetlabs/code`. + + - `master-var-dir`: Optional. The path to the OpenVox [cache directory][]. The default is `/opt/puppetlabs/server/data/puppetserver`. + + - `master-run-dir`: Optional. The path to the run directory, where the service's PID file is stored. The default is `/var/run/puppetlabs/puppetserver`. + + - `master-log-dir`: Optional. The path to the log directory. If nothing is specified, it uses the Puppet default `/var/log/puppetlabs/puppetserver`. + + - `max-active-instances`: Optional. The maximum number of JRuby instances allowed. The default is 'num-cpus - 1', with a minimum value of 1 and a maximum value of 4. In multithreaded mode, this controls the + number of threads allowed to run concurrently through the single JRuby instance. + + - `max-requests-per-instance`: Optional. The number of HTTP requests a given JRuby instance will handle in its lifetime. When a JRuby instance reaches this limit, it is flushed from memory and replaced with a + fresh one. The default is 0, which disables automatic JRuby flushing. + + JRuby flushing can be useful for working around buggy module code that would otherwise cause memory leaks, but it slightly reduces performance whenever a new JRuby instance reloads all of the Puppet + Ruby code. If memory leaks from module code are not an issue in your deployment, the default value of 0 performs best. + + - `multithreaded`: Optional, false by default. Configures OpenVox Server to use a single JRuby instance to process requests that require a JRuby, processing a number of threads up to `max-active-instances` at + a time. Reduces the memory footprint of the server by only requiring a single JRuby. + + - `max-queued-requests`: Optional. The maximum number of requests that may be queued waiting to borrow a JRuby from the pool. When this limit is exceeded, a 503 "Service Unavailable" response will be returned + for all new requests until the queue drops below the limit. If `max-retry-delay` is set to a positive value, then the 503 responses will include a `Retry-After` header indicating a random sleep time after + which the client may retry the request. The default is 0, which disables the queue limit. + + - `max-retry-delay`: Optional. Sets the upper limit for the random sleep set as a `Retry-After` header on 503 responses returned when `max-queued-requests` is enabled. A value of 0 will cause the + `Retry-After` header to be omitted. Default is 1800 seconds which corresponds to the default run interval of the OpenVox agent. + + - `borrow-timeout`: Optional. The timeout in milliseconds, when attempting to borrow an instance from the JRuby pool. The default is 1200000. + + - `environment-class-cache-enabled`: Optional. Used to control whether the OpenVox Server service maintains a cache in conjunction with the use of the + [`environment_classes` API](./puppet-api/v3/environment_classes.html). + + If this setting is set to `true`, OpenVox Server maintains the cache. It also returns an Etag header for each GET request to the API. For subsequent GET requests that use the prior Etag value in an + If-None-Match header, when the class information available for an environment has not changed, OpenVox Server returns an HTTP 304 (Not Modified) response with no body. + + If this setting is set to `false` or is not specified, OpenVox Server doesn't maintain a cache, an Etag header is not returned for GET requests, and the If-None-Match header for an incoming request is + ignored. It therefore parses the latest available code for an environment from disk on every incoming request. + + For more information, see the [`environment_classes` API documentation][]. + + - `compile-mode`: Used to control JRuby's "CompileMode", which may improve performance. A value of `jit` enables JRuby's + "just-in-time" compilation of Ruby code. A value of `force` causes JRuby to attempt to pre-compile all Ruby code. + Can also be set to `off`. `jit` is the default. + + - `profiling-mode`: Optional. Used to enable JRuby's profiler for service startup and set it to one of the supported modes. The default value is `off`, but it can be set to one of `api`, `flat`, `graph`, + `html`, `json`, `off`, and `service`. See [ruby-prof](https://ruby-prof.github.io/reports/) for details on what the various modes do. + + - `profiler-output-file`: Optional. Used to set the output file to direct JRuby profiler output. Should be a fully qualified path writable by the service user. If not set will default to a random name inside + the service working directory. + +- The `profiler` settings configure profiling: + - `enabled`: If this is set to `true`, OpenVox Server enables profiling for the OpenVox Ruby code. The default is `true`. + +- The `puppet-admin` section configures OpenVox Server's administrative API. + + > **Note:** The `puppet-admin` setting and `client-whitelist` parameter are deprecated in favor of authorization methods. For details, see the [`auth.conf` documentation][]. + + - `authorization-required` determines whether a client certificate is required to access the endpoints in this API. If set to `false`, all requests will be permitted to access this API. If set to `true`, only + the clients whose certnames are included in the `client-whitelist` setting are allowed access to the admin API. If this setting is not specified but the `client-whitelist` setting is specified, the default + value for this setting is `true`. + + - `client-whitelist` contains an array of client certificate names that are allowed to access the admin API. OpenVox Server denies any requests made to this endpoint that do not present a valid client + certificate mentioned in this array. + + If neither the `authorization-required` nor the `client-whitelist` settings are specified, OpenVox Server uses the new authorization methods and [`auth.conf`][`auth.conf` documentation] formats to access the + admin API endpoints. + +- The `versioned-code` settings configure commands required to use [static catalogs][]: + - `code-id-command`: the path to an executable script that OpenVox Server invokes to generate a `code_id`. When compiling a static catalog, OpenVox Server uses the output of this script as the catalog's + `code_id`. The `code_id` associates the catalog with the compile-time version of any [file resources][file resource] that has a `source` attribute with a `puppet:///` URI value. + + - `code-content-command` contains the path to an executable script that OpenVox Server invokes when an agent makes a [`static_file_content`][] API request for the contents of a [file resource][] that has a + `source` attribute with a `puppet:///` URI value. + +> **Note:** The OpenVox Server process must be able to execute the `code-id-command` and `code-content-command` scripts, and the scripts must return valid content to standard output and an error code of 0. For +> more information, see the [static catalogs][] and [`static_file_content` API][`static_file_content`] documentation. +> +> If you're using static catalogs, you **must** set and use **both** `code-id-command` and `code-content-command`. If only one of those settings are specified, OpenVox Server fails to start. If neither setting +> is specified, OpenVox Server defaults to generating catalogs without static features even when an agent requests a static catalog, which the agent will process as a normal catalog. + +### Examples + + # Configuration for the JRuby interpreters. + + jruby-puppet: { + ruby-load-path: [/opt/puppetlabs/puppet/lib/ruby/vendor_ruby] + gem-home: /opt/puppetlabs/server/data/puppetserver/jruby-gems + gem-path: [/opt/puppetlabs/server/data/puppetserver/jruby-gems, /opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems] + environment-vars: { "FOO" : ${FOO} + "LANG" : "de_DE.UTF-8" } + master-conf-dir: /etc/puppetlabs/puppet + master-code-dir: /etc/puppetlabs/code + master-var-dir: /opt/puppetlabs/server/data/puppetserver + master-run-dir: /var/run/puppetlabs/puppetserver + master-log-dir: /var/log/puppetlabs/puppetserver + max-active-instances: 1 + max-requests-per-instance: 0 + } + + # Settings related to HTTP client requests made by Puppet Server. + # These settings only apply to client connections using the Puppet::Network::HttpPool + # classes. Client connections using net/http or net/https directly will not be + # configured with these settings automatically. + http-client: { + # A list of acceptable protocols for making HTTP requests + #ssl-protocols: [TLSv1, TLSv1.1, TLSv1.2] + + # A list of acceptable cipher suites for making HTTP requests. For more info on available cipher suites, see: + # http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#SunJSSEProvider + #cipher-suites: [TLS_RSA_WITH_AES_256_CBC_SHA256, + # TLS_RSA_WITH_AES_256_CBC_SHA, + # TLS_RSA_WITH_AES_128_CBC_SHA256, + # TLS_RSA_WITH_AES_128_CBC_SHA] + + # The amount of time, in milliseconds, that an outbound HTTP connection + # will wait for data to be available before closing the socket. If not + # defined, defaults to 20 minutes. If 0, the timeout is infinite and if + # negative, the value is undefined by the application and governed by the + # system default behavior. + #idle-timeout-milliseconds: 1200000 + + # The amount of time, in milliseconds, that an outbound HTTP connection will + # wait to connect before giving up. Defaults to 2 minutes if not set. If 0, + # the timeout is infinite and if negative, the value is undefined in the + # application and governed by the system default behavior. + #connect-timeout-milliseconds: 120000 + + # Whether to enable http-client metrics; defaults to 'true'. + #metrics-enabled: true + } + + # Settings related to profiling the puppet Ruby code. + profiler: { + enabled: true + } + + # Settings related to static catalogs. These paths are examples. There are no default + # scripts provided with Puppet Server, and no default path for the scripts. To use static catalog features, you must set + # the paths and provide your own scripts. + versioned-code: { + code-id-command: /opt/puppetlabs/server/apps/puppetserver/code-id-command_script.sh + code-content-command: /opt/puppetlabs/server/apps/puppetserver/code-content-command_script.sh + } diff --git a/docs/_openvox-server_9x/config_file_web-routes.markdown b/docs/_openvox-server_9x/config_file_web-routes.markdown new file mode 100644 index 000000000..f1d81dfef --- /dev/null +++ b/docs/_openvox-server_9x/config_file_web-routes.markdown @@ -0,0 +1,27 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: web-routes.conf" +--- + +The `web-routes.conf` file configures the OpenVox Server `web-router-service`, which sets mount points for OpenVox Server's web applications. +You should not modify these mount points, as OpenVox agents rely on OpenVox Server mounting them to specific URLs. + +For an overview, see [OpenVox Server Configuration](./configuration.html). To configure the `webserver` service, see the [`webserver.conf` documentation](./config_file_webserver.html). + +## Example + +The `web-routes.conf` file looks like this: + +```text +# Configure the mount points for the web apps. +web-router-service: { + # These two should not be modified because the Puppet 4 agent expects them to + # be mounted at these specific paths. + "puppetlabs.services.ca.certificate-authority-service/certificate-authority-service": "/puppet-ca" + "puppetlabs.services.master.master-service/master-service": "/puppet" + "puppetlabs.services.legacy-routes.legacy-routes-service/legacy-routes-service": "" + + # This controls the mount point for the Puppet administration API. + "puppetlabs.services.puppet-admin.puppet-admin-service/puppet-admin-service": "/puppet-admin-api" +} +``` diff --git a/docs/_openvox-server_9x/config_file_webserver.markdown b/docs/_openvox-server_9x/config_file_webserver.markdown new file mode 100644 index 000000000..1a76fe866 --- /dev/null +++ b/docs/_openvox-server_9x/config_file_webserver.markdown @@ -0,0 +1,45 @@ +--- +layout: default +title: "OpenVox Server Configuration Files: webserver.conf" +--- + +The `webserver.conf` file configures the OpenVox Server `webserver` service. For an overview, see [OpenVox Server Configuration](./configuration.html). +To configure the mount points for the OpenVox administrative API web applications, see the [`web-routes.conf` documentation](./config_file_web-routes.html). + +## Examples + +The `webserver.conf` file looks something like this: + +```text +# Configure the webserver. +webserver: { + # Log webserver access to a specific file. + access-log-config: /etc/puppetlabs/puppetserver/request-logging.xml + # Require a valid certificate from the client. + client-auth: want + # Listen for HTTPS traffic on all available hostnames. + ssl-host: 0.0.0.0 + # Listen for HTTPS traffic on port 8140. + ssl-port: 8140 +} +``` + +These are the main values for managing an OpenVox Server installation. For further documentation, including a complete list of available settings and values, see +[Configuring the Webserver Service](https://github.com/OpenVoxProject/trapperkeeper-webserver/blob/main/doc/jetty-config.md). + +By default, OpenVox Server is configured to use the correct OpenVox Server and certificate authority (CA) +certificates. If you're using an intermediate CA and providing your own certificates and keys, make sure the +SSL-related parameters in `webserver.conf` point to the correct file. + +```text +webserver: { + ... + ssl-cert : /etc/puppetlabs/puppet/ssl/certs/.pem + ssl-key : /etc/puppetlabs/puppet/ssl/private_keys/.pem + ssl-ca-cert : /etc/puppetlabs/puppetserver/ca/ca_crt.pem + ssl-cert-chain : /etc/puppetlabs/puppetserver/ca/ca_crt.pem + ssl-crl-path : /etc/puppetlabs/puppet/ssl/crl.pem +} +``` + +Configuring an intermediate CA requires additional steps, which are described in [Intermediate CA](./intermediate_ca.html). diff --git a/docs/_openvox-server_9x/config_logging_advanced.markdown b/docs/_openvox-server_9x/config_logging_advanced.markdown new file mode 100644 index 000000000..77c4e6ef5 --- /dev/null +++ b/docs/_openvox-server_9x/config_logging_advanced.markdown @@ -0,0 +1,143 @@ +--- +layout: default +title: "OpenVox Server: Advanced Logging Configuration" +--- + +OpenVox Server uses the [Logback](http://logback.qos.ch/) library to handle all of its logging. Logback configuration settings are stored in the [`logback.xml`](./config_file_logbackxml.html) file, which is +located at `/etc/puppetlabs/puppetserver/logback.xml` by default. + +You can configure Logback to log messages in JSON format, which makes it easy to send them to other logging backends, such as Logstash. + +## Configuring OpenVox Server for use with Logstash + +There are a few steps necessary to setup your OpenVox Server logging for use with Logstash. The first step is to modify your logging configuration so that OpenVox Server is logging in a JSON format. After that, +you'll configure an external tool to monitor these JSON files and send the data to Logstash (or another remote logging system). + +### Configuring OpenVox Server to log to JSON + +Before you configure OpenVox Server to log to JSON, consider the following: + +- Do you want to configure OpenVox Server to _only_ log to JSON, instead of the default plain-text logging? Or do you want to have JSON logging _in addition to_ the default plain-text logging? +- Do you want to set up JSON logging _only_ for the main OpenVox Server logs (`puppetserver.log`), or _also_ for the HTTP access logs (`puppetserver-access.log`)? +- What kind of log rotation strategy do you want to use for the new JSON log files? + +The following examples show how to configure Logback for: + +- logging to both JSON and plain-text +- JSON logging both the main logs and the HTTP access logs +- log rotation on the JSON log files + +Adjust the example configuration settings to suit your needs. + +> **Note:** OpenVox Server also relies on Logback to manage, rotate, and archive Server log files. Logback archives Server logs when they exceed 200MB, and when the total size of all Server logs exceeds 1GB, it +> automatically deletes the oldest logs. + +#### Adding a JSON version of the main OpenVox Server logs + +Logback writes logs using components called [appenders](http://logback.qos.ch/manual/appenders.html). The example code below uses `RollingFileAppender` to rotate the log files and avoid consuming all of your +storage. + +1. To configure OpenVox Server to log its main logs to a second log file in JSON format, add an appender section like the following example to your `logback.xml` file, at the same level in the XML as existing + appenders. The order of the appenders does not matter. + + ```xml + + /var/log/puppetlabs/puppetserver/puppetserver.log.json + + + /var/log/puppetlabs/puppetserver/puppetserver.log.json.%d{yyyy-MM-dd} + 5 + + + + + ``` + +2. Activate the appended by adding an `appender-ref` entry to the `` section of `logback.xml`: + + ```xml + + + + + ``` + +3. If you decide you want to log _only_ the JSON format, comment out the other `appender-ref` entries. + +`LogstashEncoder` has many configuration options, including the ability to modify the list of fields that you want to include, or give them different field names. For more information, see the +[Logstash Logback Encoder Docs](https://github.com/logstash/logstash-logback-encoder/blob/master/README.md#loggingevent-fields). + +#### Adding a JSON version of the OpenVox Server HTTP Access logs + +To add JSON logging for HTTP requests: + +1. Add the following Logback appender section to the `request-logging.xml` file: + + ```xml + {% raw %} + + /var/log/puppetlabs/puppetserver/puppetserver-access.log.json + + + /var/log/puppetlabs/puppetserver/puppetserver-access.log.json.%d{yyyy-MM-dd} + 30 + + + + + + + + { + "@timestamp":"%date{yyyy-MM-dd'T'HH:mm:ss.SSSXXX}", + "clientip":"%remoteIP", + "auth":"%user", + "verb":"%requestMethod", + "requestprotocol":"%protocol", + "rawrequest":"%requestURL", + "response":"#asLong{%statusCode}", + "bytes":"#asLong{%bytesSent}", + "total_service_time":"#asLong{%elapsedTime}", + "request":"http://%header{Host}%requestURI", + "referrer":"%header{Referer}", + "agent":"%header{User-agent}", + + "request.host":"%header{Host}", + "request.accept":"%header{Accept}", + "request.accept-encoding":"%header{Accept-Encoding}", + "request.connection":"%header{Connection}", + + "puppet.client-verify":"%header{X-Client-Verify}", + "puppet.client-dn":"%header{X-Client-DN}", + "puppet.client-cert":"%header{X-Client-Cert}", + + "response.content-type":"%responseHeader{Content-Type}", + "response.content-length":"%responseHeader{Content-Length}", + "response.server":"%responseHeader{Server}", + "response.connection":"%responseHeader{Connection}" + } + + + + + + {% endraw %} + ``` + +2. Add a corresponding `appender-ref` in the `configuration` section: + + ```xml + + ``` + +For more information about options available for the `pattern` section, see the [Logback Logstash Encoder Docs](https://github.com/logstash/logstash-logback-encoder/blob/master/README.md#accessevent-fields). + +### Sending the JSON data to Logstash + +After configuring OpenVox Server to log messages in JSON format, you must also configure it to send the logs to Logstash (or another external logging system). There are several different ways to approach this: + +- Configure Logback to send the data to Logstash directly, from within OpenVox Server. See the Logstash-Logback encoder docs on how to send the logs by + [TCP](https://github.com/logstash/logstash-logback-encoder/blob/master/README.md#tcp) or [UDP](https://github.com/logstash/logstash-logback-encoder/blob/master/README.md#udp). Note that TCP comes with the + risk of bottlenecking OpenVox Server if your Logstash system is busy, and UDP might silently drop log messages. +- [Filebeat](https://www.elastic.co/products/beats/filebeat) is a tool from Elastic for shipping log data to Logstash. +- [Logstash Forwarder](https://github.com/elastic/logstash-forwarder) is an earlier tool from Elastic with similar capabilities. diff --git a/docs/_openvox-server_9x/configuration.markdown b/docs/_openvox-server_9x/configuration.markdown new file mode 100644 index 000000000..4fa4e4461 --- /dev/null +++ b/docs/_openvox-server_9x/configuration.markdown @@ -0,0 +1,169 @@ +--- +layout: default +title: "OpenVox Server Configuration" +--- + +[deprecated]: ./deprecated_features.html + +OpenVox Server automatically loads the settings in the `main`, `master` and `server` sections of the configuration file. +If there are duplicates, it prefers the values in the `server` section. OpenVox Server honors the +following `puppet.conf` settings: + +- allow_duplicate_certs +- autosign +- cacert +- cacrl +- cakey +- ca_name +- capub +- ca_ttl +- certdir +- certname +- cert_inventory +- codedir (PE only) +- csrdir +- csr_attributes +- dns_alt_names +- hostcert +- hostcrl +- hostprivkey +- hostpubkey +- keylength +- localcacert +- manage_internal_file_permissions +- privatekeydir +- requestdir +- serial +- signeddir +- ssl_client_header +- ssl_client_verify_header +- trusted_oid_mapping_file + +However, for some tasks, such as configuring the web server or an external Certificate Authority (CA), OpenVox Server has separate configuration files and settings. These files and settings are described below. +For more information about differences between OpenVox Server and the Ruby OpenVox Server's use of `puppet.conf` settings, see +[OpenVox Server: Differing Behavior in `puppet.conf`](./puppet_conf_setting_diffs.html). + +## Configuration Files + +OpenVox Server's configuration files and settings (with the exception of the [logging config file](#logging)) are in the `conf.d` directory, located by default at `/etc/puppetlabs/puppetserver/conf.d`. These +config files are in the HOCON format, which keeps the basic structure of JSON but is more readable. For more information, see the +[HOCON documentation](https://github.com/typesafehub/config/blob/master/HOCON.md). + +At startup, OpenVox Server reads all the `.conf` files in the `conf.d` directory. You must restart OpenVox Server for any changes to those files to take effect. The `conf.d` directory contains the following files +and settings: + +- [`global.conf`](./config_file_global.html) +- [`webserver.conf`](./config_file_webserver.html) +- [`web-routes.conf`](./config_file_web-routes.html) +- [`puppetserver.conf`](./config_file_puppetserver.html) +- [`auth.conf`](./config_file_auth.html) +- [`ca.conf`](./config_file_ca.html) +- [`master.conf`](./config_file_master.html) ([deprecated][]) + +The [`product.conf`](./config_file_product.html) file is optional and is not included by default. You can create that file in the `conf.d` directory in order to configure product-related settings, such as +automatic update checking and analytics data collection. + +## Logging + +OpenVox Server's logging is routed through the JVM [Logback](http://logback.qos.ch/) library. The default Logback configuration file is at `/etc/puppetserver/logback.xml` or +`/etc/puppetlabs/puppetserver/logback.xml`. You can edit this file to change the logging behavior, or specify a different Logback config file in [`global.conf`](./config_file_global.html). + +For more information on the `logback.xml` file, see [its documentation](./config_file_logbackxml.html) and the [Logback documentation](http://logback.qos.ch/manual/configuration.html). For advanced logging +configuration tips, such as configuring Logstash or outputting logs in JSON format, see [the Advanced Logging Configuration guide](./config_logging_advanced.html). + +For some tips on advanced logging configuration, including information about configuring your system to write logs in a JSON format suitable for sending to logstash or other external logging systems, see the +[Advanced Logging Configuration](./config_logging_advanced.html) documentation. + +### HTTP Traffic + +OpenVox Server logs HTTP traffic in a format similar to Apache, and to a separate file than the main log file. By default, this is located at `/var/log/puppetlabs/puppetserver/puppetserver-access.log`. + +By default, the following information is logged for each HTTP request: + +- remote host +- remote log name +- remote user +- date of the logging event +- URL requested +- status code of the request +- response content length +- remote IP address +- local port +- elapsed time to serve the request, in milliseconds + +The Logback configuration file is at `/etc/puppetlabs/puppetserver/request-logging.xml`. You can edit this file to change the logging behavior. Specify a different Logback configuration file in +[`webserver.conf`](./config_file_webserver.html) with the [`access-log-config`](https://github.com/openvoxproject/trapperkeeper-webserver/blob/main/doc/jetty-config.md#access-log-config) setting. For +more information on configuring the logged data, see [Logback Access Pattern Layout](http://logback.qos.ch/manual/layouts.html#AccessPatternLayout). + +### Authorization + +To enable additional logging related to `auth.conf`, edit OpenVox Server's `logback.xml` file. By default, only a single message is logged when a request is denied. + +To enable a one-time logging of the parsed and transformed `auth.conf` file, add the following to OpenVox Server's `logback.xml` file: + +```text + +``` + +To enable rule-by-rule logging for each request as it's checked for authorization, add the following to OpenVox Server's `logback.xml` file: + +```text + +``` + +## Service Bootstrapping + +OpenVox Server is built on top of our open-source Clojure application framework, [Trapperkeeper](https://github.com/openvoxproject/trapperkeeper). + +One of the features that Trapperkeeper provides is the ability to enable or disable individual services that an application provides. In OpenVox Server, you can use this feature to enable or disable the CA +service. The CA service is enabled by default, but if you're running a multi-master environment or using an external CA, you might want to disable the CA service on some nodes. + +The service bootstrap configuration files are in two locations: + +- `/etc/puppetlabs/puppetserver/services.d/`: For services that users are expected to manually configure if necessary, such as CA-related settings. +- `/opt/puppetlabs/server/apps/puppetserver/config/services.d/`: For services users _shouldn't_ need to configure. + +Any files with a `.cfg` extension in either of these locations are combined to form the final set of services OpenVox Server will use. + +The CA-related configuration settings are set in `/etc/puppetlabs/puppetserver/services.d/ca.cfg`. If services added in future versions have user-configurable settings, the configuration files will also be in +this directory. When upgrading OpenVox Server with a package manager, it should not overwrite files already in this directory. + +In the `ca.cfg` file, find and modify these lines as directed to enable or disable the service: + +```text +# To enable the CA service, leave the following line uncommented +puppetlabs.services.ca.certificate-authority-service/certificate-authority-service +# To disable the CA service, comment out the above line and uncomment the line below +#puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service +``` + +## Adding Java JARs + +You can provide Java JARs to be loaded upon OpenVox Server's startup. + +When launched, OpenVox Server automatically loads any JARs placed in `/opt/puppetlabs/server/data/puppetserver/jars` into the `classpath`. +JARs placed here will not be modified or removed when upgrading OpenVox Server. + +## Enabling the Insecure SSLv3 Protocol + +OpenVox Server usually cannot use SSLv3, because it is disabled by default at the JRE layer. (As of javase 7u75 / 1.7.0_u75. See the +[7u75 Update Release Notes](http://www.oracle.com/technetwork/java/javase/7u75-relnotes-2389086.html) for more information.) + +You should almost always leave SSLv3 disabled, because it is compromized by the [POODLE vulnerability](https://en.wikipedia.org/wiki/POODLE) and no longer secure. +If you have clients that can't use newer protocols, you should try to upgrade them instead of downgrading OpenVox Server. + +However, if you absolutely must, you can allow OpenVox Server to negotiate with SSLv3 clients. + +To enable SSLv3 at the JRE layer, first create a properties file (for example, `/etc/sysconfig/puppetserver-properties/java.security`) with the following content: + +```text +# Override properties in $JAVA_HOME/jre/lib/security/java.security +# An empty value enables all algorithms including INSECURE SSLv3 +# java should be started with +# -Djava.security.properties=/etc/sysconfig/puppetserver-properties/java.security +# for this file to take effect. +jdk.tls.disabledAlgorithms= +``` + +After this property file exists, update `JAVA_ARGS`, typically defined in `/etc/sysconfig/puppetserver`, and add the option `-Djava.security.properties=/etc/sysconfig/puppetserver-properties/java.security`. +This will configure the JVM to override the `jdk.tls.disabledAlgorithms` property defined in `$JAVA_HOME/jre/lib/security/java.security`. Restart the `puppetserver` service for this setting to take effect. diff --git a/docs/_openvox-server_9x/crl_refresh.markdown b/docs/_openvox-server_9x/crl_refresh.markdown new file mode 100644 index 000000000..c63ffbbea --- /dev/null +++ b/docs/_openvox-server_9x/crl_refresh.markdown @@ -0,0 +1,21 @@ +--- +layout: default +title: "OpenVox Server: Automatic CRL refresh" +--- + +OpenVox Server can automatically reload an updated CRL into the running SSL context, so that the revocation of an agent's certificate no longer requires a restart of the service to take effect. +Revocation is now transparent and requires no service downtime. + +If you are upgrading and have modified your ca.cfg, adding the following line manually may be required. +See [Service Bootstraping](./configuration.html#service-bootstrapping) for information on how to update your OpenVox Server's services bootstrap configuration. + +`puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service` + +## Implementation + +Automatic CRL refresh leverages the the [trapperkeeper file system watcher](https://github.com/openvoxproject/trapperkeeper-filesystem-watcher) to watch for changes to the CRL file, and loads the updated +CRL on change. + +### Contributors + +Thanks to Jeremy Barlow, who laid the groundwork for this feature in OpenVox Server. diff --git a/docs/_openvox-server_9x/deprecated_features.markdown b/docs/_openvox-server_9x/deprecated_features.markdown new file mode 100644 index 000000000..66125ec2a --- /dev/null +++ b/docs/_openvox-server_9x/deprecated_features.markdown @@ -0,0 +1,285 @@ +--- +layout: default +title: "OpenVox Server: Deprecated Features" +--- + +The following features / configuration settings are deprecated and will be removed in a future major release of OpenVox Server. + +## `certificate-status` settings + +### Now + +If the `certificate-authority.certificate-status.authorization-required` setting is `false`, all requests that are successfully validated by SSL (if applicable for the port settings on the server) are permitted +to use the [Certificate Status](https://github.com/openvoxproject/openvox/blob/master/api/docs/http_certificate_status.md) HTTP API endpoints. This includes requests which do not provide an SSL client certificate. + +If the `certificate-authority.certificate-status.authorization-required` setting is `true` or not specified and the `puppet-admin.client-whitelist` setting has one or more entries, only the requests whose +Common Name in the SSL client certificate subject matches one of the `client-whitelist` entries are permitted to use the certificate status HTTP API endpoints. + +For any other configuration, requests are only permitted to access the certificate status HTTP API endpoints if allowed per the rule definitions in the `trapperkeeper-authorization` "auth.conf" file. See the +[puppetserver "auth.conf"](./config_file_auth.html) page for more information. + +### In a Future Major Release + +The `certificate-status` settings will be ignored completely by OpenVox Server. Requests made to the `certificate-status` HTTP API will only be allowed per the `trapperkeeper-authorization` "auth.conf" +configuration. + +### Detecting and Updating + +Look at the `certificate-status` settings in your configuration. If `authorization-required` is set to `false` or `client-whitelist` has one or more entries, these settings would be used to authorize access to +the certificate status HTTP API instead of `trapperkeeper-authorization`. + +If `authorization-required` is set to `true` or is not specified and if the `client-whitelist` was empty, you could just remove the `certificate-authority` section from your configuration. The only behavior +that would change in OpenVox Server from doing this would be that a warning message would no longer be written to the "puppetserver.log" file at startup. + +If `authorization-required` is set to `false`, you would need to create a corresponding rule in the `trapperkeeper-authorization` file which would allow unauthenticated client access to the certificate status +API. + +For example: + +```hocon +authorization: { + version: 1 + rules: [ + { + match-request: { + path: "/certificate_status/" + type: path + method: [ get, put, delete ] + } + allow-unauthenticated: true + sort-order: 200 + name: "certificate_status" + }, + { + match-request: { + path: "/certificate_statuses/" + type: path + method: get + } + allow-unauthenticated: true + sort-order: 200 + name: "certificate_statuses" + }, + ... + ] +} +``` + +If `authorization-required` is set to `true` or not set but the `client-whitelist` has one or more custom entries in it, you would need to create a corresponding rule in the `trapperkeeper-authorization` +"auth.conf" file which would allow only specific clients access to the certificate status API. + +For example, the current certificate status configuration could have: + +```hocon +certificate-authority: + certificate-status: { + client-whitelist: [ admin1, admin2 ] + } +} +``` + +Corresponding `trapperkeeper-authorization` rules could have: + +```hocon +authorization: { + version: 1 + rules: [ + { + match-request: { + path: "/certificate_status/" + type: path + method: [ get, put, delete ] + } + allow: [ admin1, admin2 ] + sort-order: 200 + name: "certificate_status" + }, + { + match-request: { + path: "/certificate_statuses/" + type: path + method: get + } + allow: [ admin1, admin2 ] + sort-order: 200 + name: "certificate_statuses" + }, + ... + ] +} +``` + +After adding the desired rules to the `trapperkeeper-authorization` "auth.conf" file, remove the `certificate-authority` section from the "puppetserver.conf" file and restart the puppetserver service. + +### Context + +In previous OpenVox Server releases, there was no unified mechanism for controlling access to the various endpoints that OpenVox Server hosts. OpenVox Server used core OpenVox "auth.conf" to authorize requests +handled by its core API endpoints, and custom client whitelists for the CA and Admin endpoints. The custom client whitelists do not provide granular enough control to meet some use cases. + +`trapperkeeper-authorization` unifies authorization configuration across all of these endpoints into a single file and provides more granular control. + +## `puppet-admin` Settings + +### Now + +If the `puppet-admin.authorization-required` setting is `false`, all requests that are successfully validated by SSL (if applicable for the port settings on the server) are permitted to use the `puppet-admin` +HTTP API endpoints. This includes requests which do not provide an SSL client certificate. + +If the `puppet-admin.authorization-required` setting is `true` or not specified and the `puppet-admin.client-whitelist` setting has one or more entries, only the requests whose Common Name in the SSL client +certificate subject matches one of the `client-whitelist` entries are permitted to use the `puppet-admin` HTTP API endpoints. + +For any other configuration, requests are only permitted to access the `puppet-admin` HTTP API endpoints if allowed per the rule definitions in the `trapperkeeper-authorization` "auth.conf" file. See the +[puppetserver "auth.conf"](./config_file_auth.html) page for more information. + +### In a Future Major Release + +The `puppet-admin` settings will be ignored completely by OpenVox Server. Requests made to the `puppet-admin` HTTP API will only be allowed per the `trapperkeeper-authorization` "auth.conf" configuration. + +### Detecting and Updating + +Look at the `puppet-admin` settings in your configuration. If `authorization-required` is set to `false` or `client-whitelist` has one or more entries, these settings would be used to authorize access to the +`puppet-admin` HTTP API instead of `trapperkeeper-authorization`. + +If `authorization-required` is set to `true` or is not specified and if the `client-whitelist` was empty, you could just remove the `puppet-admin` section from your configuration and restart your puppetserver +service in order for OpenVox Server to start using the `trapperkeeper-authorization` "auth.conf" file. The only behavior that would change in OpenVox Server from doing this would be that a warning message would +no longer be written to the puppetserver.log file. + +If `authorization-required` is set to `false`, you would need to create corresponding rules in the `trapperkeeper-authorization` file which would allow unauthenticated client access to the "puppet-admin" API +endpoints. + +For example: + +```hocon +authorization: { + version: 1 + rules: [ + { + match-request: { + path: "/puppet-admin-api/v1/environment-cache" + type: path + method: delete + } + allow-unauthenticated: true + sort-order: 200 + name: "environment-cache" + }, + { + match-request: { + path: "/puppet-admin-api/v1/jruby-pool" + type: path + method: delete + } + allow-unauthenticated: true + sort-order: 200 + name: "jruby-pool" + }, + ... + ] +} +``` + +If `authorization-required` is set to `true` or not set but the `client-whitelist` has one or more custom entries in it, you would need to create corresponding rules in the `trapperkeeper-authorization` +"auth.conf" file which would allow only specific clients access to the "puppet-admin" API endpoints. + +For example, the current "puppet-admin" configuration could have: + +```hocon +puppet-admin: { + client-whitelist: [ admin1, admin2 ] +} +``` + +Corresponding `trapperkeeper-authorization` rules could have: + +```hocon +authorization: { + version: 1 + rules: [ + { + match-request: { + path: "/puppet-admin-api/v1/environment-cache" + type: path + method: delete + } + allow: [ admin1, admin2 ] + sort-order: 200 + name: "environment-cache" + }, + { + match-request: { + path: "/puppet-admin-api/v1/jruby-pool" + type: path + method: delete + } + allow: [ admin1, admin2 ] + sort-order: 200 + name: "jruby-pool" + }, + ... + ] +} +``` + +After adding the desired rules to the `trapperkeeper-authorization` "auth.conf" file, remove the `puppet-admin` section from the "puppetserver.conf" file and restart the puppetserver service. + +### Context + +In previous OpenVox Server releases, there was no unified mechanism for controlling access to the various endpoints that OpenVox Server hosts. OpenVox Server used core OpenVox "auth.conf" to authorize requests +handled by its core API endpoints, and custom client whitelists for the CA and Admin endpoints. The custom client whitelists do not provide granular enough control to meet some use cases. + +`trapperkeeper-authorization` unifies authorization configuration across all of these endpoints into a single file and provides more granular control. + +## `master.allow-header-cert-info` setting + +### Now + +The `master.allow-header-cert-info` setting in [`puppetserver.conf`](./config_file_puppetserver.html) controls whether OpenVox Server reads client identity, such as the client's certificate name, from HTTP headers +instead of from the SSL connection. This is used when OpenVox Server runs behind a TLS-terminating proxy or load balancer that authenticates the client certificate and forwards the details in headers. + +OpenVox Server also accepts an equivalent `allow-header-cert-info` setting in the `authorization` section of the [`trapperkeeper-authorization` "auth.conf"](./config_file_auth.html) file. If +`master.allow-header-cert-info` is set, OpenVox Server logs a deprecation warning. + +### In a Future Major Release + +The `master.allow-header-cert-info` setting will be ignored completely. Whether OpenVox Server trusts certificate information from HTTP headers will be controlled only by the +`authorization.allow-header-cert-info` setting. + +### Detecting and Updating + +Look for `allow-header-cert-info` in the `master` section of your `puppetserver.conf` file. Set `allow-header-cert-info` in the `authorization` section of your `trapperkeeper-authorization` "auth.conf" file to +the same value, then remove the setting from the `master` section. + +Only enable `allow-header-cert-info` when OpenVox Server is behind a trusted TLS-terminating proxy that sets these headers. +If it is enabled when requests can reach OpenVox Server directly, clients can spoof their identity through the headers. +{: .warning } + +### Context + +Header-based certificate information was historically configured separately for the legacy request handler and for `trapperkeeper-authorization`. Consolidating on the `authorization` setting keeps all +request-authorization configuration in the `trapperkeeper-authorization` "auth.conf" file. + +## `jruby-puppet` `master-*` directory settings + +### Now + +The `jruby-puppet` section of [`puppetserver.conf`](./config_file_puppetserver.html) accepts both the `server-*` directory settings and their older `master-*` equivalents. The `master-*` settings are deprecated: + +| Deprecated setting | Replacement | +| --- | --- | +| `master-conf-dir` | `server-conf-dir` | +| `master-code-dir` | `server-code-dir` | +| `master-var-dir` | `server-var-dir` | +| `master-run-dir` | `server-run-dir` | +| `master-log-dir` | `server-log-dir` | + +### In a Future Major Release + +The `master-*` directory settings will be removed. Use the `server-*` settings instead. + +### Detecting and Updating + +Look in the `jruby-puppet` section of your `puppetserver.conf` file for any of the `master-*` directory settings listed above. Rename each to its `server-*` equivalent; the values do not change. + +### Context + +These settings were renamed as part of moving away from "master" terminology toward "server". The `server-*` names are the supported form. diff --git a/docs/_openvox-server_9x/dev_debugging.markdown b/docs/_openvox-server_9x/dev_debugging.markdown new file mode 100644 index 000000000..61c463fcf --- /dev/null +++ b/docs/_openvox-server_9x/dev_debugging.markdown @@ -0,0 +1,95 @@ +--- +layout: default +title: "OpenVox Server: Debugging" +--- + +Because OpenVox Server executes both Clojure and Ruby code, approaches to debugging differ depending on which part of the application you're interested in. + +## Debugging Clojure Code + +If you are interested in debugging the web service layer or other parts of the app that are written in Clojure, there are lots of options available. The Clojure REPL is often the most useful tool, as it makes +it very easy to interact with individual functions and namespaces. + +If you are looking for more traditional debugging capabilities, such as defining breakpoints and stepping through the lines of your source code, there are many options. Just about any Java debugging tool will +work to some degree, but Clojure-specific tools such as [CDT](http://georgejahad.com/clojure/cdt.html) and [debug-repl](http://github.com/georgejahad/debug-repl) will have better integration with your Clojure +source files. + +For a more full-featured IDE, [Cursive](https://cursiveclojure.com/) is a great option. It's built on [IntelliJ IDEA](http://www.jetbrains.com/idea/), and provides a debug REPL that supports all of the same +debugging features that are available in Java; breakpoints, evaluating expressions in the local scope when stopped at a breakpoint, visual navigation of the call stack across threads, etc. + +## Debugging Ruby Code + +Debugging the Ruby code running in OpenVox Server can be a bit trickier, because Java and Clojure debugging tools will only take you into the JRuby interpreter source code, not into the Ruby code that it is +processing. So, if you wish to debug the Ruby code directly, you'll need to install gems and take advantage of their capabilities (not unlike how you would debug Ruby code in the MRI interpreter). + +For more info on installing gems for OpenVox Server, see [OpenVox Server and Gems](./gems.html). + +## Ruby REPL incompatible with Lein REPL + +Please note that a REPL running in Ruby is incompatible with `lein repl` because JRuby will not receive data from standard input when running inside of `lein repl`. To use a ruby REPL during development run +`puppetserver` from source with `lein run` rather than `lein repl`: + + lein run --config ~/.puppetserver/puppetserver.conf + +The `lein run` command will start the server in the foreground as normal. `pry` or `ruby-debug` will display an input prompt when the relevant statement is reached. Expect to see the normal `lein run` output +and then the Ruby REPL will present itself as compared to `lein repl` which presents a prompt early in the process lifecycle. In this way the "ruby repl" is more of a breakpoint than a REPL in the Clojure +sense. + +## `ruby-debug` + +### Installation + +There are many gems available that provide various ways of debugging Ruby code depending on what version of Ruby and which Ruby interpreter you're running. One of the most common gems is `ruby-debug`, and there +is a JRuby-compatible version available. To install it for use in OpenVox Server, run: + + sudo puppetserver gem install ruby-debug + +Or, if you're running puppetserver from source: + + lein gem -c /path/to/puppetserver.conf install ruby-debug + +### Usage + +After installing the gem, you can trigger the debugger by adding a line like this to any of the Ruby code that is run in OpenVox Server (including the OpenVox Ruby code): + + require 'ruby-debug'; debugger + +## `pry` + +### Installation + +Pry is another popular gem for introspecting Ruby code. It is compatible with JRuby. Install `pry` when running a packaged version of puppetserver using: + + sudo puppetserver gem install pry --no-ri --no-rdoc + +Or, if you're running puppetserver from source: + + $ lein gem -c ~/puppetserver/puppetserver.conf -- install pry \ + --no-ri --no-rdoc + +### Usage + +`puppetserver` should be run in the foreground to make use of the pry repl. This involves stopping the background service and starting the server in the foreground with the `puppet foreground` subcommand: + + sudo service puppetserver stop + sudo puppetserver foreground + +After installing, you can add a line like this to the Ruby code: + + require 'pry'; binding.pry + +This will give you an advanced interactive REPL at the line of code where you've called pry. + +There are many other gems that are useful for debugging, and a large percentage of them are compatible with JRuby. If you have a favorite that is not mentioned here please let us know, and we will consider +adding it to this documentation! + +## Limitations + +We are aware that some favorite gems/tools/features for ruby debugging don't currently work with JRuby/OpenVox Server. (For example, some things like color syntax highlighting in Pry.) +It's important to us to make sure that the Ruby developer experience is not degraded for developers working via OpenVox Server, so, if you run into issues like this, please file an issue on our +[Bug Tracker](https://github.com/openvoxproject), and we will see if it's possible to add support for things that we're missing. In many cases it might be a matter of simply submitting a patch to JRuby, +or submitting a JRuby-compatibility patch for an existing gem, and we're interested in trying to help with those sorts of things whenever possible. + +# Tracing Code Events + +OpenVox Server can utilize JRuby's standard facilities for tracing events during code execution. For more information on these techniques, see the [Tracing Code Events](./dev_trace_func.html) page. diff --git a/docs/_openvox-server_9x/dev_running_from_source.markdown b/docs/_openvox-server_9x/dev_running_from_source.markdown new file mode 100644 index 000000000..660403aa7 --- /dev/null +++ b/docs/_openvox-server_9x/dev_running_from_source.markdown @@ -0,0 +1,360 @@ +--- +layout: default +title: "Running OpenVox Server from Source" +--- + +## So you'd like to run OpenVox Server from source? + +The following steps will help you get OpenVox Server up and running from source. + +## Quick Start for Developers + +This quick start assumes the [prerequisites](#step-1-install-prerequisites) are already installed: Java and Leiningen to run the server, Git to check out the source, and Ruby 3.1 or later to run the +agent. See [Step 1](#step-1-install-prerequisites) for details. + +```console +# clone git repository and initialize submodules +$ git clone --recursive https://github.com/OpenVoxProject/openvox-server +$ cd openvox-server + +# Move any old config aside (rename rather than delete, so you can restore +# it) if you want to be sure you're starting from the latest defaults +$ mv ~/.puppetlabs ~/.puppetlabs.bak.$(date +%Y%m%d-%H%M%S) + +# Run the `dev-setup` script to initialize all required configuration +$ ./dev-setup + +# Launch the clojure REPL +$ lein repl +# Run Puppet Server +dev-tools=> (go) +dev-tools=> (help) +``` + +You should now have a running server. All relevant paths (`$confdir`, `$codedir`, etc.) are configured by default to point to directories underneath `~/.puppetlabs`. These should all align with the default +values that `puppet` uses (for non-root users). + +You can find the specific paths in the `dev/puppetserver.conf` file. + +In another shell, you can run the agent from source. Run it through Bundler so the source tree is isolated from any `puppet` or `openvox` gems installed in your Ruby (otherwise the two copies of +Puppet collide): + +```console +# Go to the agent source in your openvox-server checkout +$ cd openvox-server/ruby/puppet +# Install the source tree's dependencies (first time only) +$ bundle install +# Run the source agent against your running server +$ bundle exec puppet agent -t --confdir ~/.puppetlabs/etc/puppet +``` + +More detailed instructions follow. + +## Step 1: Install Prerequisites + +Use your system's package tools to ensure that the following prerequisites are installed: + +- Java 21 or 25 +- [Leiningen 2.12.0 or later](http://leiningen.org/) +- Git (for checking out the source code) +- Ruby 3.2 or later -- only needed to run a Puppet Agent from source on the host (the source agent is run through Bundler, which isolates it from any Puppet gems in your Ruby). You can skip this if you + [run the agent in a Docker container](#running-the-agent-inside-a-docker-container) instead. + +## Step 2: Clone Git Repo and Set Up Working Tree + +```console +git clone --recursive https://github.com/OpenVoxProject/openvox-server +cd openvox-server +``` + +## Step 3: Set up Config Files + +The easiest way to do this is to just run: + +```console +./dev-setup +``` + +This will set up all of the necessary configuration files and directories inside of your `~/.puppetlabs` directory. If you are interested in seeing what all of the default file paths are, you can find them in +`./dev/puppetserver.conf`. + +The default paths should all align with the default values that are used by `puppet` (for non-root users). + +If you'd like to customize your environment, here are a few things you can do: + +- Before running `./dev-setup`, set an environment variable called `MASTERHOST`. If this variable is found during `dev-setup`, it will configure your `puppet.conf` file to use this value for your certname (both + for Puppet Server and for `puppet`) and for the `server` configuration (so that your agent runs will automatically use this hostname as their Puppet Server). +- After running `./dev-setup`, you can enable autosigning so that agent certificate requests are signed automatically -- convenient when [running agents in Docker + containers](#running-the-agent-inside-a-docker-container). Configure this before you start the server in Step 4 so the change is picked up without a restart. It is best suited to development + environments rather than production: + + ```console + puppet config set autosign true --section server + ``` + +- Create a file called `dev/user.clj`. This file will be automatically loaded when you run Puppet Server from the REPL. In it, you can define a function called `get-config`, and use it to override the default + values of various settings from `dev/puppetserver.conf`. For an example of what this file should look like, see `./dev/user.clj.sample`. + +You don't need to create a `user.clj` in most cases; settings most likely to warrant it are: + +- `jruby-puppet.max-active-instances`: the number of JRuby instances to put into the pool. This can usually be set to 1 for dev purposes, unless you're working on something that involves concurrency. +- `jruby-puppet.splay-instance-flush`: Do not attempt to splay JRuby flushing, set when testing if using multiple JRuby instances and you need to control when they are flushed from the pool +- `jruby-puppet.server-conf-dir`: the OpenVox Server confdir (where `puppet.conf`, `modules`, `manifests`, etc. should be located). +- `jruby-puppet.server-code-dir`: the OpenVox Server codedir +- `jruby-puppet.server-var-dir`: the OpenVox Server vardir +- `jruby-puppet.server-run-dir`: the OpenVox Server rundir +- `jruby-puppet.server-log-dir`: the OpenVox Server logdir + +## Step 4a: Run the server from the clojure REPL + +The preferred way of running the server for development purposes is to run it from inside the clojure REPL. The git repo includes some files in the `/dev` directory that are intended to make this process +easier. + +When running a clojure REPL via the `lein repl` command-line command, lein will load the `dev/dev-tools.clj` namespace by default. + +Running the server inside of the clojure REPL allows you to make changes to the source code and reload the server without having to restart the entire JVM. It can be much faster than running from the command +line, when you are doing iterative development. We are also starting to build up a library of utility functions that can be used to inspect and modify the state of the running server; see `dev/dev-tools.clj` +for more info. + +(NOTE: many of the developers of this project are using a more full-featured IDE called [Cursive Clojure](https://cursiveclojure.com/), built on the IntelliJ IDEA platform, for our daily development. It +contains an integrated REPL that can be used in place of the `lein repl` command-line command, and works great with all of the functions described in this document.) + +To start the server from the REPL, run the following: + +```clojure +$ lein repl +nREPL server started on port 47631 on host 127.0.0.1 +dev-tools=> (go) +dev-tools=> (help) +``` + +Then, if you make changes to the source code, all you need to do in order to restart the server with the latest changes is: + +```clojure +dev-tools=> (reset) +``` + +Restarting the server this way should be significantly faster than restarting the entire JVM process. + +You can also run the utility functions to inspect the state of the server, e.g.: + +```clojure +dev-tools=> (print-puppet-environment-states) +``` + +Have a look at `dev-tools.clj` if you're interested in seeing what other utility functions are available. + +## Step 4b: Run the server from the command line + +If you prefer not to run the server interactively in the REPL, you can launch it as a normal process. To start the OpenVox Server when running from source, simply run the following: + +```console +lein run -c /path/to/puppetserver.conf +``` + +## Step 4c: Development environment gotchas + +### Missing git submodules + +If you get an error like the following: + +```text +Execution error (LoadError) at org.jruby.RubyKernel/require +(org/jruby/RubyKernel.java:970). +(LoadError) no such file to load -- puppet +``` + +Then you've probably forgotten to fetch the git submodules. + +### Failing tests + +If you change the `:webserver :ssl-port` config option from the default value of `8140`, tests will fail with errors like the following: + +```text +lein test :only puppetlabs.general-puppet.general-puppet-int-test/test-external-command-execution + +ERROR in (test-external-command-execution) (SocketChannelImpl.java:-2) +Uncaught exception, not in assertion. +expected: nil +2019-02-06 14:58:50,541 WARN [async-dispatch-18] [o.e.j.s.h.ContextHandler] Empty contextPath + actual: java.net.ConnectException: Connection refused + at sun.nio.ch.SocketChannelImpl.checkConnect (SocketChannelImpl.java:-2) + sun.nio.ch.SocketChannelImpl.finishConnect (SocketChannelImpl.java:717) + org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent (DefaultConnectingIOReactor.java:171) + org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents (DefaultConnectingIOReactor.java:145) + org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute (AbstractMultiworkerIOReactor.java:348) + org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute (PoolingNHttpClientConnectionManager.java:192) + org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run (CloseableHttpAsyncClientBase.java:64) + java.lang.Thread.run (Thread.java:844) +``` + +Changing the `ssl-port` variable back to `8140` makes the tests run properly. + +## Running the Agent + +Use a command like the one below to run an agent against your running OpenVox Server: + +```console +puppet agent --confdir ~/.puppetlabs/etc/puppet \ + --debug -t +``` + +Note that a system installed Puppet Agent is ok for use with source-based OpenVoxDB and OpenVox Server. The `--confdir` above specifies the same confdir that OpenVox Server is using. Because the Puppet Agent and +OpenVox Server instances are both using the same confdir, they're both using the same certificates as well. This alleviates the need to sign certificates as a separate step. + +To run the agent from source instead -- for example, to exercise local changes to the agent code -- use the Bundler approach shown in [Quick Start for Developers](#quick-start-for-developers). + +## Running the Agent inside a Docker container + +You can run a Puppet Agent inside a Docker container to test against an OpenVox Server you are running from source. Unlike the host agent described above, a containerized agent has its own certificate +identity, so it must connect to the server by the **hostname** the server certificate was issued for. Connecting by the Docker host IP (for example `--server 172.17.0.1`) fails with a TLS hostname +mismatch, because the server certificate is issued for a name rather than an address. + +### Prerequisites + +Before starting the server (Step 4), configure it as described in [Step 3: Set up Config Files](#step-3-set-up-config-files): + +- Run `MASTERHOST=puppet ./dev-setup` so the server certificate covers the `puppet` hostname the agent connects to. +- Enable autosigning so each new agent certificate is signed automatically, rather than signing each one by hand. + +If you would rather sign certificates manually, see [Sign the agent certificate](#sign-the-agent-certificate) below. + +### Start the agent + +The agent image already defaults to the server name `puppet`, so you only need to make that name resolve to your Docker host. Map it with `--add-host`, and persist the agent's SSL directory in a +named volume so it keeps a stable key and certificate across runs: + +```console +docker run -ti \ + --name agent1 \ + --add-host puppet:host-gateway \ + -v agent1-ssl:/etc/puppetlabs/puppet/ssl \ + ghcr.io/openvoxproject/openvoxagent:8 \ + agent -t --certname agent1 +``` + +- `--add-host puppet:host-gateway` resolves the `puppet` hostname inside the container to your Docker host, where the server you are running from source listens on port `8140`. +- `-v agent1-ssl:/etc/puppetlabs/puppet/ssl` persists the agent's keys and certificates in a named volume. Without it, each `docker run` generates a new private key, causing certificate/key + mismatches on subsequent runs. + +On Linux you can instead share the host's network stack with `--network host --add-host puppet:127.0.0.1`, which points the `puppet` hostname at the loopback address the server is bound to. + +### Sign the agent certificate + +With autosigning enabled in the [Prerequisites](#prerequisites), each agent certificate is signed automatically and you can skip this step. + +If you turn autosigning off, the agent's first run submits a certificate signing request and then exits without compiling a catalog (`Certificate for agent1 has not been signed yet`). Sign the +pending request on the server, then run the agent again. From source, use the `puppetserver-ca` CLI and point it at your development config: + +```console +puppetserver-ca sign --certname agent1 --config ~/.puppetlabs/etc/puppet/puppet.conf +``` + +The CLI signs by connecting to the running server under the name in that config (`puppet`), so that name must also resolve on the host running the command -- for example, with a `127.0.0.1 puppet` +entry in `/etc/hosts`. Because the SSL directory is persisted in the named volume, the agent reuses its signed certificate on every subsequent run. + +### Re-running a container + +To start a previously created agent container again, use the `docker start` command: + +```console +docker start -a agent1 +``` + +## Running tests + +- `lein test` to run the clojure test suite +- `rake spec` to run the jruby test suite + +The Clojure test suite can consume a lot of transient memory. Using a larger JVM heap size when running tests can significantly improve test run time. The default heap size is somewhat conservative: 1 GB for +the minimum heap (much lower than that as a maximum can lead to Java OutOfMemory errors during the test run) and 2 GB for the maximum heap. While the heap size can be configured via the `-Xms` and `-Xmx` +arguments for the `:jvm-opts` `defproject` key within the `project.clj` file, it can also be customized for an individual user environment via either of the following methods: + +1. An environment variable named `PUPPETSERVER_HEAP_SIZE`. For example, to use a heap size of 6 GiB for a `lein test` run, you could run the following: + + ```console + PUPPETSERVER_HEAP_SIZE=6G lein test + ``` + +2. A lein `profiles.clj` setting in the `:user` profile under the `:puppetserver-heap-size` key. For example, to use a heap size of 6 GiB, you could add the following key to your `~/.lein/profiles.clj` file: + + ```clojure + {:user {:puppetserver-heap-size "6G" + ...}} + ``` + +With the `:puppetserver-heap-size` key defined in the `profiles.clj` file, any subsequent `lein test` run would utilize the associated value for the key. If both the environment variable and the `profiles.clj` +key are defined, the value from the environment variable takes precedence. When either of these settings is defined, the value is used as both the minimum and maximum JVM heap size. + +From anecdotal testing, at least a heap size of 5 GB provides the best performance benefit for full runs of the Clojure unit test suite. This value may change over time depending upon how the tests evolve. + +## Installing Ruby Gems for Development + +The gems that are vendored with the openvox-server OS packages will be automatically installed into your dev environment by the `./dev-setup` script. If you wish to install additional gems, please see the +[Gems](./gems.html) document for detailed information. + +## Debugging + +For more information about debugging both Clojure and JRuby code, please see [OpenVox Server: Debugging](./dev_debugging.html) documentation. + +## Running OpenVoxDB + +To run a source OpenVoxDB with OpenVox Server, OpenVox Server needs standard OpenVoxDB configuration and how to find the OpenVoxDB terminus. First copy the +`dev/puppetserver.conf` file to another directory. In your copy of the config, append a new entry to the `ruby-load-path` list: `/puppet/lib`. This tells +OpenVox Server to load the OpenVoxDB terminus from the specified directory. + +From here, the instructions are similar to installing OpenVoxDB manually via packages. The OpenVox Server instance needs configuration for connecting to OpenVoxDB. See the +[OpenVoxDB documentation](/openvoxdb/latest/connect_puppet_server.html) for details. + +Update `~/.puppetlabs/etc/puppet/puppet.conf` to include: + +```ini +[server] +storeconfigs = true +storeconfigs_backend = puppetdb +reports = store,puppetdb +``` + +Create a new puppetdb config file `~/.puppetlabs/etc/puppet/puppetdb.conf` that contains + +```ini +[main] +server_urls = https://:8081 +``` + +Then create a new routes file at `~/.puppetlabs/etc/puppet/routes.yaml` that contains + +```yaml +--- +server: + facts: + terminus: puppetdb + cache: yaml +``` + +Assuming you have an OpenVoxDB instance up and running, start your OpenVox Server instance with the new puppetserver.conf file that you changed: + +```console +lein run -c ~//puppetserver.conf +``` + +Depending on your OpenVoxDB configuration, you might need to change some SSL config. OpenVoxDB requires that the same CA that signs its certificate also has signed OpenVox Server's certificate. The easiest way +to do this is to point OpenVoxDB at the same configuration directory that OpenVox Server and Puppet Agent are pointing to. Typically this setting is specified in the `jetty.ini` file in the OpenVoxDB conf.d +directory. The update would look like: + +```ini +[jetty] + +#... +ssl-cert = /.puppetlabs/etc/puppet/ssl/certs/.pem +ssl-key = /.puppetlabs/etc/puppet/ssl/private_keys/.pem +ssl-ca-cert = /.puppetlabs/etc/puppet/ssl/certs/ca.pem +``` + +After the SSL config is in place, start (or restart) OpenVoxDB: + +```console +lein run services -c /conf.d +``` + +Then run the Puppet Agent and you should see activity in OpenVoxDB and OpenVox Server. diff --git a/docs/_openvox-server_9x/dev_trace_func.markdown b/docs/_openvox-server_9x/dev_trace_func.markdown new file mode 100644 index 000000000..79d4071a8 --- /dev/null +++ b/docs/_openvox-server_9x/dev_trace_func.markdown @@ -0,0 +1,63 @@ +--- +layout: default +title: "OpenVox Server: Tracing Code Events" +--- + + +The JRuby runtime supports the Ruby [set_trace_func](http://ruby-doc.org/core-1.9.3/Kernel.html#method-i-set_trace_func) Kernel +method for tracing code events, e.g., lines of code being executed and calls +to C-language routines or Ruby methods. This can likewise be used in OpenVox +Server for tracing. + +In order to enable a more verbose level of tracing, e.g., to capture lower-level +calls into C code, the `jruby.debug.fullTrace` Java property must be set to +"true". If you are running OpenVox Server from source, this can be done by +adding the option to the `project.clj` file: + +```ini +:jvm-opts ["-Djruby.debug.fullTrace=true"] +``` + +If you are running OpenVox Server from a package, this can be done by adding the +option to the `puppetserver` file in `/etc/sysconfig` or `/etc/default`, +depending upon your OS distribution: + +```ini +JAVA_ARGS="-Xms2g -Xmx2g -Djruby.debug.fullTrace=true" +``` + +A call to the `set_trace_func` function can be done in one of the Ruby files in +the OpenVox Server code. For the trace to be in effect for the full execution +of Ruby code, one common place to put this call would be at the top of the +`../src/ruby/puppetserver-lib/puppet/server/master.rb` file, the OpenVox Server +master class. A basic implementation might look like this: + +```ruby +set_trace_func proc { |event, file, line, id, binding, classname| + printf "%8s %s:%-2d %10s %8s\n", event, file, line, id, classname +} +``` + +Note that `printf` will write each trace line to stdout. If you are running +Puppet Server from a package install, stdout should be routed to the +`/var/log/puppetserver-daemon.log` file. + +Lines of output from `set_trace_func` look like the following: + +```text + c-call /usr/share/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/jopenssl19/openssl/ssl-internal.rb:30 initialize OpenSSL::X509::Store +``` + +You could use this technique to locate any references made to specific class +names from code and the active stack at the point of each reference. For +example, to locate callers of any `OpenSSL` classes, you could add the following +to the `set_trace_func` call: + +```ruby +set_trace_func proc { |event, file, line, id, binding, classname| + if classname.to_s =~ /OpenSSL/ + printf "%8s %s:%-2d %10s %8s\n", event, file, line, id, classname + puts caller + end +} +``` diff --git a/docs/_openvox-server_9x/external_ssl_termination.markdown b/docs/_openvox-server_9x/external_ssl_termination.markdown new file mode 100644 index 000000000..5938f002d --- /dev/null +++ b/docs/_openvox-server_9x/external_ssl_termination.markdown @@ -0,0 +1,70 @@ +--- +layout: default +title: "Puppet Server: External SSL Termination" +canonical: "/puppetserver/latest/external_ssl_termination.html" +--- + +Use the following steps to configure external SSL termination. + +## Disable HTTPS for Puppet Server + +You'll need to turn off SSL and have Puppet Server use the HTTP protocol instead: remove the `ssl-port` and `ssl-host` settings from the `conf.d/webserver.conf` file and replace them with `port` and `host` +settings. See [Configuring the Webserver Service](https://github.com/openvoxproject/trapperkeeper-webserver/blob/main/doc/jetty-config.md) for more information on configuring the web server service. + +## Allow Client Cert Data From HTTP Headers + +When using external SSL termination, Puppet Server expects to receive client certificate information via some HTTP headers. + +By default, reading this data from headers is disabled. To allow Puppet Server to recognize it, you'll need to set `allow-header-cert-info: true` in the `authorization` config section of the +`/etc/puppetlabs/puppetserver/conf.d/auth.conf` file. + +See [Puppet Server Configuration](./configuration.html) for more information on the `puppetserver.conf` and `auth.conf` files. + +Note: This assumes the default behavior of Puppet 5 and greater of using Puppet Server's hocon auth.conf rather Puppet's older ini-style auth.conf. + +> **WARNING**: Setting `allow-header-cert-info` to 'true' puts Puppet Server in an incredibly vulnerable state. Take extra caution to ensure it is **absolutely not reachable** by an untrusted network. +> +> With `allow-header-cert-info` set to 'true', authorization code will use only the client HTTP header values---not an SSL-layer client certificate---to determine the client subject name, authentication status, +> and trusted facts. This is true even if the web server is hosting an HTTPS connection. This applies to validation of the client via rules in the +> [auth.conf](/openvox/latest/config_file_auth.html) file and any [trusted facts][trusted] extracted from certificate extensions. +> +> If the `client-auth` setting in the `webserver` config block is set to `need` or `want`, the Jetty web server will still validate the client certificate against a certificate authority store, but it will only +> verify the SSL-layer client certificate---not a certificate in an `X-Client-Cert` header. + +## Reload Puppet Server + +You'll need to reload Puppet Server for the configuration changes to take effect. + +## Configure SSL Terminating Proxy to Set HTTP Headers + +The device that terminates SSL for Puppet Server must extract information from the client's certificate and insert that information into three HTTP headers. See the documentation for your SSL terminator for +details. + +The headers you'll need to set are `X-Client-Verify`, `X-Client-DN`, and `X-Client-Cert`. + +### `X-Client-Verify` + +Mandatory. Must be either `SUCCESS` if the certificate was validated, or something else if not. (The convention seems to be to use `NONE` for when a certificate wasn't presented, and `FAILED:reason` for other +validation failures.) Puppet Server uses this to authorize requests; only requests with a value of `SUCCESS` will be considered authenticated. + +### `X-Client-DN` + +Mandatory. Must be the [Subject DN][] of the agent's certificate, if a certificate was presented. Puppet Server uses this to authorize requests. + +[subject dn]: /docs/background/ssl/cert_anatomy.html#the-subject-dn-cn-certname-etc + +### `X-Client-Cert` + +Optional. Should contain the client's [PEM-formatted][pem format] (Base-64) certificate (if a certificate was presented) in a single URI-encoded string. Note that URL encoding is not sufficient; all space +characters must be encoded as `%20` and not `+` characters. + +> **Note:** Puppet Server only uses the value of this header to extract [trusted facts][trusted] from extensions in the client certificate. If you aren't using trusted facts, you can choose to reduce the size +> of the request payload by omitting the `X-Client-Cert` header. + + + +> **Note:** Apache's `mod_proxy` converts line breaks in PEM documents to spaces for some reason, and Puppet Server can't decode the result. We're tracking this issue as +> [SERVER-217](https://tickets.puppetlabs.com/browse/SERVER-217). + +[pem format]: /docs/background/ssl/cert_anatomy.html#pem-file +[trusted]: /openvox/latest/lang_facts_and_builtin_vars.html#trusted-facts diff --git a/docs/_openvox-server_9x/gems.markdown b/docs/_openvox-server_9x/gems.markdown new file mode 100644 index 000000000..1df4fcdd8 --- /dev/null +++ b/docs/_openvox-server_9x/gems.markdown @@ -0,0 +1,105 @@ +--- +layout: default +title: "Puppet Server: Using Ruby Gems" +canonical: "/puppetserver/latest/gems.html" +--- + +If you have server-side Ruby code in your modules, Puppet Server will run it via JRuby. Generally speaking, this only affects custom parser functions, types, and report processors. For the vast majority of +cases this shouldn't pose any problems because JRuby is highly compatible with vanilla Ruby. + +Puppet Server will not load gems from user specified `GEM_HOME` and `GEM_PATH` environment variables because `puppetserver` unsets `GEM_PATH` and manages `GEM_HOME`. + +> **Note:** Starting with Puppet Server 2.7.1, you can set custom Java arguments for the `puppetserver gem` command via the `JAVA_ARGS_CLI` environment variable, either temporarily on the command line or +> persistently by adding it to the sysconfig/default file. The `JAVA_ARGS_CLI` environment variable also controls the arguments used when running the `puppetserver ruby` and `puppetserver irb` +> [subcommands](./subcommands.html). See the [Server 2.7.1 release notes](https://docs.puppet.com/puppetserver/2.7/release_notes.html) for details. + +## Gems with packaged versions of Puppet Server + +The value of `GEM_HOME` when starting the puppetserver process as root using a packaged version of `puppetserver` is: + + /opt/puppetlabs/puppet/cache/jruby-gems + +This directory does not exist by default. + +## Gems when running Puppet Server from source + +The value of `GEM_HOME` when starting the puppetserver process from the project root is: + + ./target/jruby-gems + +## Gems when running Puppet Server spec tests + +The value of `GEM_HOME` when starting the puppetserver JRuby spec tests using `rake spec` from the project root is: + + ./vendor/test_gems + +This directory is automatically populated by the `rake spec` task if it does not already exist. The directory may be safely removed and it will be re-populated the next time `rake spec` is run in your working +copy. + +## Installing and removing gems + +We isolate the Ruby load paths that are accessible to Puppet Server's JRuby interpreter, so that it doesn't load any gems or other code that you have installed on your system Ruby. If you want Puppet Server to +load additional gems, use the Puppet Server-specific `gem` command to install them. For example, to install the foobar gem, use: + + sudo puppetserver gem install foobar --no-ri --no-rdoc + +The `puppetserver gem` command is simply a wrapper around the usual Ruby `gem` command, so all of the usual arguments and flags should work as expected. For example, to show your locally installed gems, run: + + puppetserver gem list + +Or, if you're running from source: + + lein gem -c ~/.puppetserver/puppetserver.conf list + +The `puppetserver gem` command also respects the running user's `~/.gemrc` file, which you can use to configure upstream sources or proxy settings. For example, consider a `.gemrc` file containing: + + --- + :sources: [ 'https://rubygems-mirror.megacorp.com', 'https://rubygems.org' ] + http_proxy: "http://proxy.megacorp.com:8888" + +This configures the listed `:sources` as the `puppetserver gem` command's upstream sources, and uses the listed `http_proxy`, which you can confirm: + + $ puppetserver gem environment | grep proxy + - "http_proxy" => "http://proxy.megacorp.com:8888" + +As with the rest of Puppet Server's configuration, we recommend managing these settings with Puppet. You can manage Puppet Server's gem dependencies with the package provider shipped in +[`puppetlabs-puppetserver_gem`](https://forge.puppet.com/puppetlabs/puppetserver_gem) module. + +> Note: If you try to load a gem before it's been installed, the agent run will fail with a `LoadError`. If this happens, reload the server after installing the gem to resolve the issue. + +## Installing gems for use with development + +When running from source, JRuby uses a `GEM_HOME` of `./target/jruby-gems` relative to the current working directory of the process. `lein gem` should be used to install gems into this location using jruby. + +NOTE: `./target/jruby-gems` is not used when running the JRuby spec tests, gems are instead automatically installed into and loaded from `./vendor/test_gems`. If you need to install a gem for use both during +development and testing make sure the gem is available in both directories. + +As an example, the following command installs `pry` locally in the project. Note the use of `--` to pass the following command line arguments to the gem script. + + $ lein gem --config ~/.puppetserver/puppetserver.conf -- install pry \ + --no-ri --no-rdoc + Fetching: coderay-1.1.0.gem (100%) + Successfully installed coderay-1.1.0 + Fetching: slop-3.6.0.gem (100%) + Successfully installed slop-3.6.0 + Fetching: method_source-0.8.2.gem (100%) + Successfully installed method_source-0.8.2 + Fetching: spoon-0.0.4.gem (100%) + Successfully installed spoon-0.0.4 + Fetching: pry-0.10.1-java.gem (100%) + Successfully installed pry-0.10.1-java + 5 gems installed + +With the gem installed into the project tree `pry` can be invoked from inside Ruby code. For more detailed information on `pry` see [Puppet Server: Debugging](./dev_debugging.html#pry). + +## Gems with Native (C) Extensions + +If, in your custom parser functions or report processors, you're using Ruby gems that require native (C) extensions, you won't be able to install these gems under JRuby. In many cases, however, there are +drop-in replacements implemented in Java. For example, the popular [Nokogiri](http://www.nokogiri.org/) gem for processing XML provides a completely compatible Java implementation that's automatically installed +if you run `gem install` via JRuby or Puppet Server, so you shouldn't need to change your code at all. + +In other cases, there may be a replacement gem available with a slightly different name; e.g., `jdbc-mysql` instead of `mysql`. The JRuby wiki +[C Extension Alternatives](https://github.com/jruby/jruby/wiki/C-Extension-Alternatives) page discusses this issue further. + +If you're using a gem that won't run on JRuby and you can't find a suitable replacement, please open a ticket on our [Issue Tracker](https://tickets.puppet.com/browse/SERVER); we're definitely interested in +helping provide solutions if there are common gems that are causing trouble for users! diff --git a/docs/_openvox-server_9x/http_api_index.markdown b/docs/_openvox-server_9x/http_api_index.markdown new file mode 100644 index 000000000..7e5eeafbf --- /dev/null +++ b/docs/_openvox-server_9x/http_api_index.markdown @@ -0,0 +1,150 @@ +--- +layout: default +title: "OpenVox Server HTTP API: Index" +--- + +OpenVox Server provides several services via HTTP API, and the OpenVox agent application uses those services to resolve a node's credentials, retrieve a configuration catalog, retrieve file data, and submit +reports. + +## V1/V2 HTTP APIs (removed) + +The V1 and V2 APIs were removed in Puppet 4.0.0. The routes that were previously under `/` or `/v2.0` can now be found under the [`/puppet/v3`](#openvox-v3-http-api) API or [`/puppet-ca/v1`](#ca-v1-http-api) +API. + +## OpenVox and OpenVox CA APIs + +OpenVox's HTTP API is split into two separately versioned APIs: + +- An API for configuration-related services +- An API for the certificate authority (CA). + +All configuration endpoints are prefixed with `/puppet`, while all CA endpoints are prefixed with `/puppet-ca`. All endpoints are explicitly versioned: the prefix is always immediately followed by a string like +`/v3` (a directory separator, the letter `v`, and the version number of the API). + +### Authorization + +Authorization for `/puppet` and `/puppet-ca` endpoints is controlled with [OpenVox Server's `auth.conf` authorization system](config_file_auth.html). + +## OpenVox V3 HTTP API + +The OpenVox agent application uses several network services to manage systems. These services are all grouped under the `/puppet` API. +Other tools can access these services and use the OpenVox Server's data for other purposes. + +The V3 API contains endpoints of two types: those that are based on dispatching to OpenVox's internal "indirector" framework, and those that are not (namely the [environments endpoint](#environments-endpoint)). + +Every HTTP endpoint that dispatches to the indirector follows the form `/puppet/v3/:indirection/:key?environment=:environment`, where: + +- `:environment` is the name of the environment that should be in effect for the request. Not all endpoints need an environment, but the query parameter must always be specified. +- `:indirection` is the indirection to which the request is dispatched. +- `:key` is the "key" portion of the indirection call. + +Using this API requires significant understanding of how OpenVox's internal services are structured, but the following documents specify what is available and how to interact with it. + +### Configuration management services + +The OpenVox agent application directly uses these services to manage the configuration of a node. + +These endpoints accept payload formats formatted as JSON by default (MIME type of `application/json`), except for `File Content` and `File Bucket File`, which always use `application/octet-stream`. + +> **Note:** Legacy PSON (MIME type of `text/pson`) is still an available format, but should be used only as a fallback for binary content. + +- [Facts](http_facts.html) +- [Catalog](http_catalog.html) +- [Node](http_node.html) +- [File bucket file](http_file_bucket_file.html) +- [File content](http_file_content.html) +- [File metadata](http_file_metadata.html) +- [Report](http_report.html) + +> **Note:** The [Puppet v4 catalog API](puppet-api/v4/catalog.html) is preferred for new integrations. It does +> not require facts to be submitted as part of the catalog request. + +### Environments endpoint + +The `/puppet/v3/environments` endpoint uses a different format than the configuration management endpoints. + +The endpoint accepts only payloads formatted as JSON, and responds with JSON (MIME type of `application/json`). + +- [Environments](http_environments.html) + +### OpenVox Server-specific endpoints + +OpenVox Server adds several unique endpoints of its own. They include these additional `/puppet/v3/` endpoints: + +- [Environment classes](puppet-api/v3/environment_classes.html), at `/puppet/v3/environment_classes` +- [Environment modules](puppet-api/v3/environment_modules.html), at `/puppet/v3/environment_modules` +- [Static file content](puppet-api/v3/static_file_content.html), at `/puppet/v3/static_file_content` + +It also includes these unique APIs, with endpoints containing other URL prefixes: + +- [Status API](status-api/v1/services.html), at `/status/v1/services` +- [Metrics v1 (mbeans) API](metrics-api/v1/metrics_api.html), at `/metrics/v1/mbeans` +- [Metrics v2 (Jolokia) API](metrics-api/v2/metrics_api.html), at `/metrics/v2/` +- Admin API, at `/puppet-admin-api/v1/`: + - [Environment cache](admin-api/v1/environment-cache.html), at `/puppet-admin-api/v1/environment-cache` + - [JRuby pool](admin-api/v1/jruby-pool.html), at `/puppet-admin-api/v1/jruby-pool` + +### Error responses + +The `environments` endpoint responds to error conditions in a uniform manner and uses standard HTTP response codes to signify those errors. + +| Request problem | HTTP API error response code | +| --------------- | ---------------------------- | +| Client submits malformed request | 400 Bad Request | +| Unauthorized client | 403 Not Authorized | +| Client uses an HTTP method not permitted for the endpoint | 405 Method Not Allowed | +| Client requests a response in a format other than JSON | 406 Unacceptable | +| Server encounters an unexpected error while handling a request | 500 Server Error | +| Server can't find an endpoint handler for an HTTP request | 404 Not Found | + +Except for HEAD requests, error responses contain a body of a uniform JSON object with the following properties: + +- `message`: (`String`) A human-readable message explaining the error. +- `issue_kind`: (`String`) A unique label to identify the error class. + +OpenVox provides a [JSON schema for error objects](/openvox/latest/schemas/error.json). Endpoints implemented by OpenVox Server have a different error schema: + +```json +{ + "msg": "", + "kind": "" +} +``` + +## CA V1 HTTP API + +The certificate authority (CA) API contains all of the endpoints supporting OpenVox's public key infrastructure (PKI) system. + +The CA V1 endpoints share the same basic format as the OpenVox V3 API, because they are based on the interface of OpenVox's indirector-based CA. +However, OpenVox Server's CA is implemented in Clojure. Both have a different prefix and version than the V3 API. + +These endpoints follow the form `/puppet-ca/v1/:indirection/:key?environment=:environment`, where: + +- `:environment` is an arbitrary placeholder word, required for historical reasons. No CA endpoints actually use an environment, but the query parameter must always be specified. +- `:indirection` is the indirection to which the request is dispatched. +- `:key` is the "key" portion of the indirection call. + +As with the OpenVox V3 API, using this API requires a significant amount of understanding of how OpenVox's internal services are structured. +The following documents specify what is available and how to interact with it. + +### SSL certificate-related services + +These endpoints accept only plain-text payload formats. Historically, OpenVox has used the MIME type `s` to mean `text/plain`. It now uses `text/plain`, but continues to accept `s` as an equivalent. + +- [Certificate](ca-api/v1/http_certificate.html) +- [Certificate Signing Requests](ca-api/v1/http_certificate_request.html) +- [Certificate Status](ca-api/v1/http_certificate_status.html) +- [Certificate Revocation List](ca-api/v1/http_certificate_revocation_list.html) +- [Certificate Clean](ca-api/v1/http_certificate_clean.html) +- [Certificate Expirations](ca-api/v1/http_certificate_expirations.html) +- [Certificate Renewal](ca-api/v1/http_certificate_renewal.html) +- [Bulk Certificate Sign](ca-api/v1/http_certificate_sign.html) + +## Serialization formats + +OpenVox sends messages using several serialization formats. Not all REST services support all of the formats. + +- [JSON](https://tools.ietf.org/html/rfc7159) +- [PSON](pson.html) (deprecated — see the PSON page for details) + +`YAML` was supported in earlier versions of OpenVox, but is no longer for security reasons. diff --git a/docs/_openvox-server_9x/http_catalog.md b/docs/_openvox-server_9x/http_catalog.md new file mode 100644 index 000000000..8aeec58d7 --- /dev/null +++ b/docs/_openvox-server_9x/http_catalog.md @@ -0,0 +1,246 @@ +--- +layout: default +title: "OpenVox Server HTTP API: Catalog (v3)" +--- + +## Catalog + +> **Note:** For new integrations, prefer the [Puppet v4 catalog API](puppet-api/v4/catalog.html). The v4 API is +> more efficient because it does not require facts to be submitted as part of the catalog request. + +The `catalog` endpoint returns a catalog for the specified node name given the provided facts. + +## Find + +Retrieve a catalog. + + POST /puppet/v3/catalog/:nodename + GET /puppet/v3/catalog/:nodename?environment=:environment + +### Supported HTTP Methods + +POST, GET + +### Supported Response Formats + +`application/json`, `text/pson` + +### Notes + +The POST and GET methods are functionally equivalent. Both provide the parameters specified below: POST sends them +in the request body, GET sends them in the query string. + +OpenVox originally used GET; POST was added because some web servers have a maximum URI length of 1024 bytes, which +is easily exceeded with the `facts` parameter. + +The examples below use the POST method. + +### Parameters + +Four parameters should be provided to the POST or GET: + +- `environment`: the environment name. +- `facts_format`: must be `application/json` or `pson`. +- `facts`: serialized JSON or PSON of the facts hash. Since facts can contain `&`, which is also the HTTP query + parameter delimiter, facts are doubly-escaped. +- `transaction_uuid`: a transaction uuid identifying the entire transaction (shows up in the report as well). + +Two optional parameters are required for static catalogs: + +- `static_catalog`: a boolean requesting a static catalog if available; should always be `true`. +- `checksum_type`: a dot-separated list of checksum types supported by the agent, for use in file resources of a + static catalog. The order signifies preference, highest first. + +Optional parameters that may be provided to the POST or GET: + +- `configured_environment`: the environment configured on the client. May be provided to notify an ENC that the + client requested a specific environment which might differ from what the client believes is its current + environment. +- `job_id`: which orchestration job triggered this catalog request. + +### Example Response + +#### Catalog found + + POST /puppet/v3/catalog/elmo.mydomain.com + + environment=env&configured_environment=canary_env&facts_format=application%2Fjson&facts=%257B%2522name%2522%253A%2522elmo.mydomain.com%2522%252C%2522values%2522%253A%257B%2522architecture%2522%253A%2522x86_64%2522%257D%257D&transaction_uuid=aff261a2-1a34-4647-8c20-ff662ec11c4c + + HTTP 200 OK + Content-Type: application/json + + { + "tags": [ + "settings", + "multi_param_class", + "class" + ], + "name": "elmo.mydomain.com", + "version": 1377473054, + "code_id": null, + "catalog_uuid": "827a74c8-cf98-44da-9ff7-18c5e4bee41e", + "catalog_format": 1, + "environment": "production", + "resources": [ + { + "type": "Stage", + "title": "main", + "tags": [ + "stage" + ], + "exported": false, + "parameters": { + "name": "main" + } + }, + { + "type": "Class", + "title": "Settings", + "tags": [ + "class", + "settings" + ], + "exported": false + }, + { + "type": "Class", + "title": "main", + "tags": [ + "class" + ], + "exported": false, + "parameters": { + "name": "main" + } + }, + { + "type": "Class", + "title": "Multi_param_class", + "tags": [ + "class", + "multi_param_class" + ], + "line": 10, + "exported": false, + "parameters": { + "one": "hello", + "two": "world" + } + }, + { + "type": "Notify", + "title": "foo", + "tags": [ + "notify", + "foo", + "class", + "multi_param_class" + ], + "line": 4, + "exported": false, + "parameters": { + "message": "One is hello, two is world" + } + } + ], + "edges": [ + { + "source": "Stage[main]", + "target": "Class[Settings]" + }, + { + "source": "Stage[main]", + "target": "Class[main]" + }, + { + "source": "Stage[main]", + "target": "Class[Multi_param_class]" + }, + { + "source": "Class[Multi_param_class]", + "target": "Notify[foo]" + } + ], + "classes": [ + "settings", + "multi_param_class" + ] + } + +#### Static Catalog found + + POST /puppet/v3/catalog/elmo.mydomain.com + + environment=env&configured_environment=canary_env&facts_format=application%2Fjson&facts=%7B%22name%22%3A%22elmo.mydomain.com%22%2C%22values%22%3A%7B%22architecture%22%3A%22x86_64%22%7D&transaction_uuid=aff261a2-1a34-4647-8c20-ff662ec11c4c&static_catalog=true&checksum_type=sha256.md5 + + HTTP 200 OK + Content-Type: application/json + + { + "tags": [ + "settings", + "multi_param_class", + "class" + ], + "name": "elmo.mydomain.com", + "version": 1377473054, + "code_id": "arbitrary_code_id_string", + "catalog_uuid": "827a74c8-cf98-44da-9ff7-18c5e4bee41e", + "catalog_format": 1, + "environment": "production", + "resources": [ ... ], + "edges": [ ... ], + "classes": [ + "settings", + "multi_param_class" + ], + "metadata": { + "/tmp/foo": { + "checksum": { + "type": "sha256", + "value": "{sha256}5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03" + }, + "content_uri": "puppet:///modules/a_module/files/foo", + "destination": null, + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/environments/production/modules/a_module/files/foo.txt", + "relative_path": null, + "source": "puppet:///modules/a_module/foo", + "type": "file" + } + }, + "recursive_metadata": { + "/tmp/bar": { + "puppet:///modules/a_module/bar": [ + { + "checksum": { + "type": "ctime", + "value": "{ctime}2016-02-19 17:38:36 -0800" + }, + "content_uri": "puppet:///modules/a_module/files/bar", + "destination": null, + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/environments/production/modules/a_module/files/bar", + "relative_path": ".", + "source": null, + "type": "directory" + } + ] + } + } + } + + +## Schema + +In the POST request body (or the GET query), the facts parameter should conform to +[the facts schema.](/openvox/latest/schemas/facts.json) + +A catalog response body conforms to +[the catalog schema.](/openvox/latest/schemas/catalog.json) diff --git a/docs/_openvox-server_9x/http_client_metrics.markdown b/docs/_openvox-server_9x/http_client_metrics.markdown new file mode 100644 index 000000000..94369fb6a --- /dev/null +++ b/docs/_openvox-server_9x/http_client_metrics.markdown @@ -0,0 +1,126 @@ +--- +layout: default +title: "OpenVox Server: HTTP Client Metrics" +--- + +[status API]: ./status-api/v1/services.html + +HTTP client metrics allow users to measure how long it takes for OpenVox Server to make requests to and receive responses from other services, such as OpenVoxDB. + +## Determining metrics IDs + +All of these metrics are of the form `puppetlabs..http-client.experimental.with-metric-id..full-response`. + +> **Note:** The `` describes what the metric measures. A metric ID is represented in the [status endpoint](./status-api/v1/services.html) as an array of strings, and in the metric itself the strings +> are joined together with periods. For instance, the metric ID of `[puppetdb resource search]` is `puppetdb.resource.search`, so the full metric name would be +> `puppetlabs..http-client.experimental.with-metric-id.puppetdb.resource.search.full-response`. + +You can configure OpenVoxDB to be a backend for configuration files (through the `storeconfigs` setting), and +you can configure OpenVox Server to send reports to an external report processing service. If you configure either of these, then during the course of handling an OpenVox agent run, OpenVox Server makes several +calls to external services to retrieve or store information. + +- During handling of a `/puppet/v3/node` request, OpenVox Server issues: + - a `facts find` request to OpenVoxDB for facts about the node, if they aren't yet cached (typically the first time it requests facts for the node). **Metric ID:** `[puppetdb facts find]`. +- During handling of a `/puppet/v3/catalog` request, OpenVox Server issues several requests: + - an OpenVoxDB `replace facts` request, to replace the facts for the agent in OpenVoxDB with the facts it received from the agent. **Metric ID:** `[puppetdb, command, replace_facts]`. + - an OpenVoxDB `resource search` request, to search for resources if exported resources are used. **Metric ID:** `[puppetdb, resource, search]`. + - an OpenVoxDB `query` request, if the `puppetdb_query` function is used in Puppet code. **Metric ID:** `[puppetdb, query]`. + - an OpenVoxDB `replace catalog` request, to replace the catalog for the agent in OpenVoxDB with the newly compiled catalog. **Metric ID:** `[puppetdb, command, replace_catalog]`. +- During handling of a `/puppet/v3/report` request, OpenVox Server issues: + - an OpenVoxDB `store report` request, to store the submitted report. **Metric ID:** `[puppetdb command store_report]`. + - a request to the configured `reports_url` to store the report, if the HTTP report processor is enabled. **Metric ID:** `[puppetdb report http]`. + +## Configuring + +HTTP client metrics are enabled by default, but can be disabled by setting `metrics-enabled` to `false` in the `http-client` section of [`puppetserver.conf`](./config_file_puppetserver.html). + +These metrics also depend on the `server-id` setting in the `metrics` section of `puppetserver.conf`. This defaults to `localhost`, and while `localhost` can collect metrics, change this setting to something +unique to avoid metric naming collisions when exporting metrics to an external tool, such as Graphite. + +This data is all available via the [status API][] endpoint, at `https://:8140/status/v1/services/master?level=debug`. If +metrics are not enabled, or if OpenVox Server has not issued any requests yet, then this array will be empty, like so: `"http-client-metrics": []`. + +In the [sample Grafana dashboard](./sample-puppetserver-metrics-dashboard.json), the `External HTTP Communications` graph visualizes all of these metrics, and the tooltip describes each of them. + +## Example metrics output + +```json +"http-client-metrics": [ + { + "aggregate": 407, + "count": 1, + "mean": 407, + "metric-id": [ + "puppetdb", + "facts", + "find" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.facts.find.full-response" + }, + { + "aggregate": 66, + "count": 1, + "mean": 66, + "metric-id": [ + "puppetdb", + "command", + "replace_facts" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.command.replace_facts.full-response" + }, + { + "aggregate": 60, + "count": 2, + "mean": 30, + "metric-id": [ + "puppetdb", + "resource", + "search" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.resource.search.full-response" + }, + { + "aggregate": 53, + "count": 1, + "mean": 53, + "metric-id": [ + "puppetdb", + "query" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.query.full-response" + }, + { + "aggregate": 22, + "count": 1, + "mean": 22, + "metric-id": [ + "puppetdb", + "command", + "store_report" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.command.store_report.full-response" + }, + { + "aggregate": 16, + "count": 1, + "mean": 16, + "metric-id": [ + "puppetdb", + "command", + "replace_catalog" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppetdb.command.replace_catalog.full-response" + }, + { + "aggregate": 2, + "count": 1, + "mean": 2, + "metric-id": [ + "puppet", + "report", + "http" + ], + "metric-name": "puppetlabs.localhost.http-client.experimental.with-metric-id.puppet.report.http.full-response" + } +], +``` diff --git a/docs/_openvox-server_9x/http_environments.md b/docs/_openvox-server_9x/http_environments.md new file mode 100644 index 000000000..d839acaa9 --- /dev/null +++ b/docs/_openvox-server_9x/http_environments.md @@ -0,0 +1,53 @@ +--- +layout: default +title: "OpenVox Server HTTP API: Environments" +--- + +## Environments + +The `environments` endpoint allows for enumeration of the environments known to the OpenVox Server. Each environment +contains information about itself such as its modulepath, manifest directory, environment timeout, and the config +version. This endpoint is by default accessible to any client with a valid certificate, though this may be changed +by `auth.conf`. + +## Get + +Get the list of known environments. + + GET /puppet/v3/environments + +### Supported Response Formats + +`application/json` + +### Parameters + +None + +### Example Request & Response + + GET /puppet/v3/environments + + HTTP 200 OK + Content-Type: application/json + + { + "search_paths": ["/etc/puppetlabs/code/environments"] + "environments": { + "production": { + "settings": { + "modulepath": ["/etc/puppetlabs/code/environments/production/modules", "/etc/puppetlabs/code/environments/development/modules"], + "manifest": ["/etc/puppetlabs/code/environments/production/manifests"] + "environment_timeout": 180, + "config_version": "/version/of/config" + } + } + } + } + +The `environment_timeout` attribute can also be the string `"unlimited"`. + +## Schema + +An environments response body conforms to +[the environments schema.](/openvox/latest/schemas/environments.json) diff --git a/docs/_openvox-server_9x/http_facts.md b/docs/_openvox-server_9x/http_facts.md new file mode 100644 index 000000000..47fe3e23b --- /dev/null +++ b/docs/_openvox-server_9x/http_facts.md @@ -0,0 +1,60 @@ +--- +layout: default +title: "OpenVox Server HTTP API: Facts" +--- + +## Facts + +The `facts` endpoint allows setting the facts for the specified node name. + +## Save + +Store facts for a node. The request body should contain JSON-formatted facts. + + PUT /puppet/v3/facts/:nodename?environment=:environment + +### Supported HTTP Methods + +PUT + +### Supported Format(s) + +`application/json`, `text/pson` + +### Parameters + +None + +### Example + +- Note: list of facts was shortened for readability. +- Note: JSON was formatted for readability. + + + + PUT /puppet/v3/facts/elmo.mydomain.com?environment=env + Content-Type: application/json + + { + "name": "elmo.mydomain.com", + "values": { + "architecture": "x86_64", + "kernel": "Darwin", + "domain": "local", + "macaddress": "70:11:24:8c:33:a9", + "osfamily": "Darwin", + "operatingsystem": "Darwin", + "facterversion": "1.7.2", + "fqdn": "elmo.mydomain.com", + }, + "timestamp": "2013-09-09 15:49:27 -0700", + "expiration": "2013-09-09 16:19:27 -0700" + } + + HTTP/1.1 200 OK + Content-Type: application/json + +## Schema + +The representation of facts contained in a PUT body should adhere to +[the facts schema.](/openvox/latest/schemas/facts.json) diff --git a/docs/_openvox-server_9x/http_file_bucket_file.md b/docs/_openvox-server_9x/http_file_bucket_file.md new file mode 100644 index 000000000..33f9a3fe4 --- /dev/null +++ b/docs/_openvox-server_9x/http_file_bucket_file.md @@ -0,0 +1,99 @@ +--- +layout: default +title: "OpenVox Server HTTP API: File Bucket File" +--- + +## File Bucket File + +The `file_bucket_file` endpoint manages the contents of files in the file bucket. All access to files is managed +with the MD5 checksum of the file contents, represented as `:md5`. Where used, `:filename` means the full absolute +path of the file on the client system. This is usually optional and used as an error check to make sure the correct +file is retrieved. The environment is required in all requests but ignored, as the file bucket does not distinguish +between environments. + +> **Note:** In OpenVox Server 9.0 and later, the default `auth.conf` lets every +> authenticated agent use `HEAD` and `PUT` on this endpoint, but `GET` and `POST` (reading +> content back out) require a client certificate with the `pp_cli_auth: "true"` extension. +> See [auth.conf](./config_file_auth.html). + +## Find + +Retrieve the contents of a file. + + GET /puppet/v3/file_bucket_file/:md5?environment=:environment + GET /puppet/v3/file_bucket_file/:md5/:original_path?environment=:environment + +This will return the contents of the file if it's present. If `:original_path` is provided then the contents will +only be sent if the file was uploaded with the same path at some point. + +## Head + +Check if a file is present in the filebucket. + + HEAD /puppet/v3/file_bucket_file/:md5?environment=:environment + HEAD /puppet/v3/file_bucket_file/:md5/:original_path?environment=:environment + +This behaves identically to find, only returning headers. + +## Save + +Save a file to the filebucket. + + PUT /puppet/v3/file_bucket_file/:md5?environment=:environment + PUT /puppet/v3/file_bucket_file/:md5/:original_path?environment=:environment + +The body should contain the file contents. This saves the file using the MD5 sum of the file contents. If +`:original_path` is provided, it adds the path to a list for the given file. If the MD5 sum in the request is +incorrect, the file will be saved under the correct checksum instead. + +### Supported HTTP Methods + +GET, HEAD, PUT + +### Supported Response Formats + +`application/octet-stream` + +### Parameters + +None + +### Examples + +#### Saving a file + + > PUT /puppet/v3/file_bucket_file/md5/eb61eead90e3b899c6bcbe27ac581660//home/user/myfile.txt?environment=production HTTP/1.1 + + > Content-Type: application/octet-stream + > Content-Length: 24 + + > This is the file content + + + < HTTP/1.1 200 OK + +#### Retrieving a file + + > GET /puppet/v3/file_bucket_file/md5/4949e56d376cc80ce5387e8e89a75396//home/user/myfile.txt?environment=production HTTP/1.1 + > Accept: application/octet-stream + + + < HTTP/1.1 200 OK + < Content-Length: 24 + + < This is the file content + +#### Wrong file name + + > GET /puppet/v3/file_bucket_file/md5/4949e56d376cc80ce5387e8e89a75396//home/user/wrong_name?environment=production HTTP/1.1 + > Accept: application/octet-stream + + + < HTTP/1.1 404 Not Found + < + < Not Found: Could not find file_bucket_file md5/4949e56d376cc80ce5387e8e89a75396/home/user/wrong_name + +## Schema + +A `file_bucket_file` response body is not structured data according to any standard scheme such as JSON or YAML, +so no schema is applicable. diff --git a/docs/_openvox-server_9x/http_file_content.md b/docs/_openvox-server_9x/http_file_content.md new file mode 100644 index 000000000..f638643fe --- /dev/null +++ b/docs/_openvox-server_9x/http_file_content.md @@ -0,0 +1,78 @@ +--- +layout: default +title: "OpenVox Server HTTP API: File Content" +--- + +## File Content + +The `file_content` endpoint returns the contents of the specified file. + +## Find + +Get a file. + + GET /puppet/v3/file_content/:mount_point/:name + +The endpoint path includes a `:mount_point` which can be one of the following types: + +- Custom file serving mounts as specified in `fileserver.conf` — see + [configuring mount points](/openvox/9.x/config_file_fileserver.html). +- `modules/` — allows access to the `files` subdirectory of `` — see + [file serving](/openvox/9.x/file_serving.html). +- `plugins` — merges the `lib` directory of every module together. Used for syncing plugins; not intended for + general consumption. Per-module sub-paths cannot be specified. +- `pluginfacts` — merges the `facts.d` directory of every module together. Used for syncing external facts; not + intended for general consumption. Per-module sub-paths cannot be specified. +- `tasks/` — allows access to files in the `tasks` subdirectory of `` — see + [file serving](/openvox/9.x/file_serving.html). + +`:name` is the path to the file within the `:mount_point` that is requested. + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`application/octet-stream` + +### Parameters + +None + +### Responses + +#### File found + + GET /puppet/v3/file_content/modules/example/my_file?environment=env + Accept: application/octet-stream + + HTTP/1.1 200 OK + Content-Type: application/octet-stream + Content-Length: 16 + + this is my file + +#### File not found + + GET /puppet/v3/file_content/modules/example/not_found?environment=env + Accept: application/octet-stream + + HTTP/1.1 404 Not Found + Content-Type: text/plain + + Not Found: Could not find file_content modules/example/not_found + +#### No file name given + + GET /puppet/v3/file_content?environment=env + + HTTP/1.1 400 Bad Request + Content-Type: text/plain + + No request key specified in /puppet/v3/file_content/ + +## Schema + +A `file_content` response body is not structured data according to any standard scheme such as JSON or YAML, +so no schema is applicable. diff --git a/docs/_openvox-server_9x/http_file_metadata.md b/docs/_openvox-server_9x/http_file_metadata.md new file mode 100644 index 000000000..e2c91af0c --- /dev/null +++ b/docs/_openvox-server_9x/http_file_metadata.md @@ -0,0 +1,373 @@ +--- +layout: default +title: "OpenVox Server HTTP API: File Metadata" +--- + +## File Metadata + +The `file_metadata` endpoint returns select metadata for a single file or many files. There are find and search +variants of the endpoint; the search variant has a trailing `s`, so it is actually `file_metadatas`. + +Although the term 'file' is used generically in the endpoint name and documentation, each returned item can be one +of the following three types: + +- File +- Directory +- Symbolic link + +The endpoint path includes a `:mount` which can be one of the following types: + +- Custom file serving mounts as specified in `fileserver.conf` — see + [configuring mount points](/openvox/9.x/config_file_fileserver.html). +- `modules/` — allows access to the `files` subdirectory of `` — see + [file serving](/openvox/9.x/file_serving.html). +- `plugins` — merges the `lib` directory of every module together. Used for syncing plugins; not intended for + general consumption. Per-module sub-paths cannot be specified. +- `pluginfacts` — merges the `facts.d` directory of every module together. Used for syncing external facts; not + intended for general consumption. Per-module sub-paths cannot be specified. +- `tasks/` — allows access to files in the `tasks` subdirectory of `` — see + [file serving](/openvox/9.x/file_serving.html). + +## Find + +Get file metadata for a single file. + + GET /puppet/v3/file_metadata/:mount/path/to/file?environment=:environment + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`application/json`, `text/pson` + +### Parameters + +Optional parameters to GET: + +- `links` — either `manage` (default) or `follow`. See examples in Search below. +- `checksum_type` — the checksum type to calculate the checksum value for the result metadata; one of `md5` + (default), `md5lite`, `sha256`, `sha256lite`, `mtime`, `ctime`, and `none`. +- `source_permissions` — whether (and how) OpenVox should copy owner, group, and mode permissions; one of: + - `ignore` (the default) will never apply the owner, group, or mode from the source when managing a file. When + creating new files without explicit permissions, the permissions they receive depend on platform-specific + behavior. On POSIX, OpenVox uses the umask of the user it is running as. On Windows, OpenVox uses the default + DACL associated with the user it is running as. + - `use` will cause OpenVox to apply the owner, group, and mode from the source to any files it is managing. + - `use_when_creating` will only apply the owner, group, and mode from the source when creating a file; existing + files will not have their permissions overwritten. + +### Example Response + +#### File metadata found for a file + + GET /puppet/v3/file_metadata/modules/example/just_a_file.txt?environment=env + + HTTP/1.1 200 OK + Content-Type: application/json + + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files/just_a_file.txt", + "relative_path": null, + "type": "file" + } + +#### File metadata found for a directory + + GET /puppet/v3/file_metadata/modules/example/subdirectory?environment=env + + HTTP/1.1 200 OK + Content-Type: application/json + + { + "checksum": { + "type": "ctime", + "value": "{ctime}2013-10-01 13:16:10 -0700" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files/subdirectory", + "relative_path": null, + "type": "directory" + } + +#### File metadata found for a link ignoring source permissions + + GET /puppet/v3/file_metadata/modules/example/link_to_file.txt?environment=env&source_permissions=ignore + + HTTP/1.1 200 OK + Content-Type: application/json + + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": "/etc/puppetlabs/code/modules/example/files/just_a_file.txt", + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files/link_to_file.txt", + "relative_path": null, + "type": "link" + } + +#### File not found + + GET /puppet/v3/file_metadata/modules/example/does_not_exist?environment=env + + HTTP/1.1 404 Not Found + + Not Found: Could not find file_metadata modules/example/does_not_exist + +## Search + +Get a list of metadata for multiple files. + + GET /puppet/v3/file_metadatas/foo.txt?environment=env + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`application/json`, `text/pson` + +### Parameters + +- `recurse` — should always be set to `yes`; unfortunately the default is `no`, which causes a search to behave + like a find operation. +- `ignore` — file or directory regex to ignore; can be repeated. +- `links` — either `manage` (default) or `follow`. See examples below. +- `checksum_type` — the checksum type to calculate the checksum value for the result metadata; one of `md5` + (default), `md5lite`, `sha256`, `sha256lite`, `mtime`, `ctime`, and `none`. +- `source_permissions` — whether (and how) OpenVox should copy owner, group, and mode permissions; see the Find + section above for the available values and their meanings. + +### Example Response + +#### Basic search + + GET /puppet/v3/file_metadatas/modules/example?environment=env&recurse=yes + + HTTP 200 OK + Content-Type: application/json + + [ + { + "checksum": { + "type": "ctime", + "value": "{ctime}2013-10-01 13:15:59 -0700" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": ".", + "type": "directory" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "just_a_file.txt", + "type": "file" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": "/etc/puppetlabs/code/modules/example/files/just_a_file.txt", + "group": 20, + "links": "manage", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "link_to_file.txt", + "type": "link" + }, + { + "checksum": { + "type": "ctime", + "value": "{ctime}2013-10-01 13:15:59 -0700" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "subdirectory", + "type": "directory" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d41d8cd98f00b204e9800998ecf8427e" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "subdirectory/another_file.txt", + "type": "file" + } + ] + +#### Search ignoring 'sub\*' and links = manage + + GET /puppet/v3/file_metadatas/modules/example?environment=env&recurse=true&ignore=sub*&links=manage + + HTTP 200 OK + Content-Type: application/json + + [ + { + "checksum": { + "type": "ctime", + "value": "{ctime}2013-10-01 13:15:59 -0700" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": ".", + "type": "directory" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": null, + "group": 20, + "links": "manage", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "just_a_file.txt", + "type": "file" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": "/etc/puppetlabs/code/modules/example/files/just_a_file.txt", + "group": 20, + "links": "manage", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "link_to_file.txt", + "type": "link" + } + ] + +#### Search ignoring "sub\*" and links = follow + +This example is identical to the above example, except for the links parameter. The resulting JSON is identical to +the above example, except for: + +- the `"links"` field is set to `"follow"` rather than `"manage"` in all metadata objects +- in the `"link_to_file.txt"` metadata: + - for `"manage"` the `"destination"` field is the link destination; for `"follow"`, it's null + - for `"manage"` the `"type"` field is `"link"`; for `"follow"` it's `"file"` + - for `"manage"` the `"mode"`, `"owner"`, and `"group"` fields are the link's values; for `"follow"` the + destination's values are used + + GET /puppet/v3/file_metadatas/modules/example?environment=env&recurse=true&ignore=sub*&links=follow + + HTTP 200 OK + Content-Type: application/json + + [ + { + "checksum": { + "type": "ctime", + "value": "{ctime}2013-10-01 13:15:59 -0700" + }, + "destination": null, + "group": 20, + "links": "follow", + "mode": 493, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": ".", + "type": "directory" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": null, + "group": 20, + "links": "follow", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "just_a_file.txt", + "type": "file" + }, + { + "checksum": { + "type": "md5", + "value": "{md5}d0a10f45491acc8743bc5a82b228f89e" + }, + "destination": null, + "group": 20, + "links": "follow", + "mode": 420, + "owner": 501, + "path": "/etc/puppetlabs/code/modules/example/files", + "relative_path": "link_to_file.txt", + "type": "file" + } + ] + + +## Schema + +The file metadata response body conforms to +[the `file_metadata` schema.](/openvox/latest/schemas/file_metadata.json) + +## Sample Module + +The examples above use this (faux) module: + + /etc/puppetlabs/code/modules/example/ + files/ + just_a_file.txt + link_to_file.txt -> /etc/puppetlabs/code/modules/example/files/just_a_file.txt + subdirectory/ + another_file.txt diff --git a/docs/_openvox-server_9x/http_node.md b/docs/_openvox-server_9x/http_node.md new file mode 100644 index 000000000..755888e9e --- /dev/null +++ b/docs/_openvox-server_9x/http_node.md @@ -0,0 +1,66 @@ +--- +layout: default +title: "OpenVox Server HTTP API: Node" +--- + +## Node + +The `node` endpoint is used by the OpenVox agent to get basic information +about a node. The returned information includes the node name and +environment, and optionally any classes set by an External Node +Classifier and a hash of parameters which may include the node's facts. +The returned node may have a different environment from the one given in +the request if OpenVox is configured with an ENC. + +## Find + +Retrieve data for a node. + + GET /puppet/v3/node/:certname?environment=:environment&transaction_uuid=:transaction_uuid&configured_environment=:environment + +### Supported HTTP Methods + +GET + +### Supported Response Formats + +`application/json`, `text/pson` + +### Parameters + +One parameter should be provided to the GET: + +- `transaction_uuid`: a transaction uuid identifying the entire transaction (shows up in the report as well) + +An optional parameter can be provided to the GET to notify a node classifier that the client requested a specific +environment, which might differ from what the client believes is its current environment: + +- `configured_environment`: the environment configured on the client + +### Examples + + > GET /puppet/v3/node/mycertname?environment=production&transaction_uuid=aff261a2-1a34-4647-8c20-ff662ec11c4c&configured_environment=production HTTP/1.1 + > Accept: application/json, text/pson + + < HTTP/1.1 200 OK + < Content-Type: application/json + < Content-Length: 4630 + + { + "name":"thinky.corp.example.net", + "parameters":{ + "architecture":"amd64", + "kernel":"Linux", + "blockdevices":"sda,sr0", + "clientversion":"3.3.1", + "clientnoop":"false", + "environment":"production", + ... + }, + "environment":"production" + } + +## Schema + +A node response body conforms to +[the node schema.](/openvox/latest/schemas/node.json) diff --git a/docs/_openvox-server_9x/http_report.md b/docs/_openvox-server_9x/http_report.md new file mode 100644 index 000000000..ad354db51 --- /dev/null +++ b/docs/_openvox-server_9x/http_report.md @@ -0,0 +1,126 @@ +--- +layout: default +title: "OpenVox Server HTTP API: Report" +--- + +## Report + +This document describes the OpenVox Server report endpoint and the schema for Report Format 9. + +The `report` endpoint allows clients to send reports to the OpenVox Server via HTTP or HTTPS. Once received by the +server they are processed by the *report processors* configured to be triggered when a report is received. For +example, storing reports in OpenVoxDB is handled by one such report processor. + +## Save + +The HTTP(S) endpoint for sending reports to the server is: + + PUT /puppet/v3/report/:nodename?environment=:environment + +### Supported HTTP Methods + +PUT + +### Supported Format(s) + +`application/json`, `text/pson` + +### Parameters + +None + +### Content + +The content of a report is typically generated by the OpenVox agent runtime and consists of a JSON serialization of +a report object, which in turn contains a structure of objects of the following types: + +- Log entries +- Metrics +- Resource statuses +- Transaction events + +This JSON serialization must comply with the endpoint's report JSON schema. + +## Example + +Here is an example of a PUT request. (Note that the content-length is not correct as the example is formatted for +readability.) + + PUT /puppet/v3/report/kermit.example.com?environment=production HTTP/1.0 + Content-Type: application/json + Content-Length: 1428 + + {"host"=>"kermit.example.com", + "time"=>"2013-09-12T03:50:59.009301000+02:00", + "configuration_version"=>1357986, + "transaction_uuid"=>"df34516e-4050-402d-a166-05b03b940749", + "code_id"=>null, + "job_id"=>null, + "catalog_uuid"=>"827a74c8-cf98-44da-9ff7-18c5e4bee41e", + "catalog_format"=>1, + "report_format"=>9, + "puppet_version"=>"5.0.0", + "status"=>"unchanged", + "transaction_completed"=>true, + "noop"=>false, + "noop_pending"=>false, + "environment"=>"test_environment", + "logs"=> + [{"level"=>"warning", + "message"=>"log message", + "source"=>"Puppet", + "tags"=>["warning"], + "time"=>"2013-09-12T03:50:59.009328000+02:00", + "file"=>nil, + "line"=>nil}], + "metrics"=> + {"resources"=> + {"name"=>"resources", + "label"=>"Resources", + "values"=> + [["total", "Total", 1], + ["skipped", "Skipped", 0], + ["failed", "Failed", 0], + ["failed_to_restart", "Failed to restart", 0], + ["restarted", "Restarted", 0], + ["changed", "Changed", 1], + ["out_of_sync", "Out of sync", 0], + ["scheduled", "Scheduled", 0]]}, + "time"=> + {"name"=>"time", + "label"=>"Time", + "values"=>[["timing", "Timing", 4], ["total", "Total", 4]]}, + "changes"=> + {"name"=>"changes", "label"=>"Changes", "values"=>[["total", "Total", 0]]}, + "events"=> + {"name"=>"events", + "label"=>"Events", + "values"=> + [["total", "Total", 0], + ["failure", "Failure", 0], + ["success", "Success", 0]]}}, + "resource_statuses"=> + {"Notify[a resource]"=> + {"title"=>"a resource", + "file"=>nil, + "line"=>nil, + "resource"=>"Notify[a resource]", + "resource_type"=>"Notify", + "provider_used"=>nil, + "containment_path"=>["Notify[a resource]"], + "evaluation_time"=>nil, + "tags"=>["notify"], + "time"=>"2013-09-12T03:50:59.009238000+02:00", + "failed"=>false, + "changed"=>true, + "out_of_sync"=>false, + "skipped"=>false, + "change_count"=>0, + "out_of_sync_count"=>0, + "events"=>[]}}, + "cached_catalog_status"=> "not_used"} + +## Schema + +The sent report objects must conform to +[the report schema.](/openvox/latest/schemas/report.json) diff --git a/docs/_openvox-server_9x/index.markdown b/docs/_openvox-server_9x/index.markdown new file mode 100644 index 000000000..c791277ec --- /dev/null +++ b/docs/_openvox-server_9x/index.markdown @@ -0,0 +1,36 @@ +--- +layout: default +title: "OpenVox Server 9" +--- + +> **OpenVox Server 9 is in prerelease.** See the [release notes](release_notes.html) for +> the current build and expect breaking changes before the stable 9.0.0 release. + +OpenVox Server is the primary server component in an OpenVox agent/server deployment. +It compiles configuration catalogs for managed nodes, serves files, manages certificates, +and receives reports from agents. +It is a Ruby and Clojure application that runs on the Java Virtual Machine (JVM). + +## How it works + +OpenVox agents periodically contact OpenVox Server over mutual-TLS HTTPS. +The server compiles a node-specific catalog from Puppet code and Hiera data, returns it +to the agent for enforcement, and collects the resulting report. +It also runs a built-in certificate authority for signing agent certificates. + +For a full description of the service and its internal components, see +[About OpenVox Server](./services_puppetserver.html). + +## OpenVox Server packages + +| Package | Contents | +| ------- | -------- | +| `openvox-server` | JVM-based catalog server; depends on `openvox-agent` 9.x. Requires Java 21 or 25 (not bundled). | + +## Getting started + +1. Review [pre-install tasks](install_pre.html) — system requirements, DNS, firewall, and time synchronization +2. [Install OpenVox Server](install_from_packages.html) +3. [Configure OpenVox Server](configuration.html) + +For community help and support resources, see the [Vox Pupuli support page](https://voxpupuli.org/openvox/support/). diff --git a/docs/_openvox-server_9x/infrastructure_crl.markdown b/docs/_openvox-server_9x/infrastructure_crl.markdown new file mode 100644 index 000000000..cae61b0f8 --- /dev/null +++ b/docs/_openvox-server_9x/infrastructure_crl.markdown @@ -0,0 +1,26 @@ +--- +layout: default +title: "Puppet Server: Infrastructure CRL" +canonical: "/puppetserver/latest/infrastructure_crl.html" +--- + +# Infrastructure certificate revocation list + +The Puppet Server CA can create a CRL that contains only revocations of those nodes that agents are expected to talk to during normal operations, for example, compile masters or hosts that agents connect to as part of agent-side functions. Puppet Server CA can distribute that CRL to agents, rather than the CRL it maintains with all node revocations. + +To create a smaller CRL, manage the content of the file at `$cadir/infra_inventory.txt`. +Provide a newline-separated list of the certnames. When revoked, they are added to the +Infra CRL. The certnames must match existing certificates issued and maintained by the +Puppet Server CA. Setting the value `certificate-authority.enable-infra-crl` to `true` +causes Puppet Server to update both its Full CRL and its Infra CRL with the certs that +match those certnames when revoked. When agents first check in, they receive a CRL that +includes only the revocations of certnames listed in the `infra_inventory.txt`. + +The infrastructure certificate revocation list is disabled by default in open source Puppet. To toggle it, update `enable-infra-crl` in the `certificate-authority` section of `puppetserver.conf`. + +This feature is disabled by default because the definition of what constitutes an +"infrastructure" node is site-specific and sites with a standard, single master +configuration have no need for the additional work. After having enabled the feature, if +you want to go back, remove the explicit setting and reload Puppet Server to turn the +default off; then, when agents first check, they receive the Full CRL as before (including +any infrastructure nodes that were revoked while the feature was enabled). diff --git a/docs/_openvox-server_9x/install_from_packages.markdown b/docs/_openvox-server_9x/install_from_packages.markdown new file mode 100644 index 000000000..5033073a8 --- /dev/null +++ b/docs/_openvox-server_9x/install_from_packages.markdown @@ -0,0 +1,73 @@ +--- +layout: default +title: "Install OpenVox Server" +--- + +[prereqs]: ./install_pre.html +[tuning]: ./tuning_guide.html +[install_linux]: /openvox/latest/install_linux.html +[openvoxdb]: /openvoxdb/latest/install_via_module.html +[downloads_page]: https://voxpupuli.org/openvox/install/ + +Install `openvox-server` on the host that will act as the Puppet CA and catalog +compiler for your infrastructure. + +**Before you begin:** Review the [pre-install tasks][prereqs] to confirm your +platform is supported, your Java version is correct, and port 8140 is open. + +1. Enable the OpenVox repository for your distribution. + + On apt-based systems, download and install the release package for your OS from + [apt.voxpupuli.org](https://apt.voxpupuli.org). For example, on Ubuntu 22.04: + + ```bash + wget https://apt.voxpupuli.org/openvox9-release-ubuntu22.04.deb + sudo dpkg -i openvox9-release-ubuntu22.04.deb + sudo apt update + ``` + + On yum/dnf-based systems, install the release package for your OS from + [yum.voxpupuli.org](https://yum.voxpupuli.org). For example, on EL 9: + + ```bash + sudo rpm -Uvh https://yum.voxpupuli.org/openvox9-release-el-9.noarch.rpm + ``` + + For the full list of supported distributions and release packages, see the + [Installing OpenVox][downloads_page] page. + +2. Install the package. + + On apt-based systems: + + ```bash + sudo apt install openvox-server + ``` + + On yum/dnf-based systems: + + ```bash + sudo yum install openvox-server + ``` + +3. Start and enable the service. + + ```bash + sudo systemctl start puppetserver + sudo systemctl enable puppetserver + ``` + +4. Verify the installation. + + ```bash + puppetserver --version + sudo systemctl status puppetserver + ``` + + The service should be active and you should see a version string printed. + +## What to do next + +- [Install OpenVox agent on Linux][install_linux] — roll out agents to managed nodes. +- [Install OpenVoxDB][openvoxdb] (optional) — enables enhanced queries and reports about your infrastructure. +- Review the [tuning guide][tuning] if you need to adjust the default 2 GB JVM heap size. diff --git a/docs/_openvox-server_9x/install_pre.md b/docs/_openvox-server_9x/install_pre.md new file mode 100644 index 000000000..e6f202504 --- /dev/null +++ b/docs/_openvox-server_9x/install_pre.md @@ -0,0 +1,79 @@ +--- +layout: default +title: "Installing OpenVox Server: Before you begin" +--- + +[architecture]: /openvox/latest/architecture.html + +Before installing OpenVox Server: + +1. Confirm your platform is supported. + + OpenVox Server is a Linux-only service. Packages are available from the OpenVox + repositories at [apt.voxpupuli.org](https://apt.voxpupuli.org) and + [yum.voxpupuli.org](https://yum.voxpupuli.org). + + **apt-based systems:** + + | Distribution | Versions | + | ------------ | -------- | + | Debian | 13 | + | Ubuntu | 22.04, 24.04, 26.04 | + + **yum/dnf-based systems:** + + | Distribution | Versions | Architectures | + | ------------ | -------- | ------------- | + | EL (RHEL, AlmaLinux, Rocky Linux, CentOS) | 8, 9, 10 | x86_64, aarch64 | + | Amazon Linux | 2023 | x86_64, aarch64 | + | Fedora | 43, 44 | x86_64, aarch64 | + | SLES | 15 | x86_64 | + | SLES | 16 | x86_64, aarch64 | + | RHEL FIPS | 8, 9 | x86_64 | + + OpenVox Server 9 no longer ships packages for Debian 11 and 12 (they only provide + Java 17) or for Amazon Linux 2. For the list the build system works from, see + [Supported platforms](/openvox/latest/supported_platforms.html). + +2. Verify your Java version. + + OpenVox Server 9 requires Java 21 or 25; Java 17 is no longer supported. Install a + supported JDK from your distribution's repositories before installing the OpenVox + Server package. OpenVox Server does not bundle a JDK. + + The `openvox-server` 9.x package depends on `openvox-agent` 9.x, so the package + manager installs or upgrades the agent on the server host along with it. + +3. Plan memory allocation. + + OpenVox Server is configured to use 2 GB of RAM by default. Make sure the host + has enough available memory. For testing on a VM, you can reduce this to 512 MB + after installation — see the [tuning guide](./tuning_guide.html). + +4. Open the required port. + + OpenVox agents connect to the server on TCP port **8140**. Make sure this port is + reachable from all managed nodes. If you are using a firewall, open it before + starting the service. + +5. Verify DNS. + + Starting in OpenVox 9, agents no longer fall back to the hostname `puppet` by + default, so plan to set the `server` setting in `puppet.conf` on each agent + explicitly. Forward and reverse DNS should be correct for the server and every + node. + +6. Synchronize clocks. + + OpenVox uses SSL certificates with time-based validity. If the clocks on the + server and agent nodes differ by more than a few minutes, certificate validation + will fail and agents will be unable to connect. Make sure NTP or a similar time + synchronization service is running on all nodes before deploying. + +7. Install and validate OpenVox Server before rolling out agents. + + In an agent-server deployment, the server must be running and reachable before + agents can check in. See the [architecture overview][architecture] for background + on deployment models. + +Once you have completed these checks, continue with [Install OpenVox Server](./install_from_packages.html). diff --git a/docs/_openvox-server_9x/install_what_and_where.md b/docs/_openvox-server_9x/install_what_and_where.md new file mode 100644 index 000000000..d4f91129a --- /dev/null +++ b/docs/_openvox-server_9x/install_what_and_where.md @@ -0,0 +1,80 @@ +--- +layout: default +title: "What gets installed and where (OpenVox Server)" +--- + +When you install the `openvox-server` package, it places files and directories in the following locations. + +## Executables and binaries + +| Location | Description | +| -------- | ----------- | +| `/opt/puppetlabs/bin/puppetserver` | Main `puppetserver` executable | +| `/opt/puppetlabs/bin/puppet` | Puppet CLI (installed by the `openvox-agent` dependency) | +| `/opt/puppetlabs/server/bin/` | `puppetserver` and `puppetdb` binaries — not on `PATH` by default | +| `/opt/puppetlabs/puppet/bin/` | Private Ruby runtime and agent binaries | + +## Configuration files + +OpenVox Server uses two configuration directories: + +**Server configuration** (`/etc/puppetlabs/puppetserver/`): + +| File | Description | +| ---- | ----------- | +| `conf.d/puppetserver.conf` | Main server settings | +| `conf.d/webserver.conf` | Jetty web server settings | +| `conf.d/web-routes.conf` | Web application mount points | +| `conf.d/auth.conf` | HTTPS access control rules | +| `conf.d/ca.conf` | Certificate authority service settings | +| `conf.d/global.conf` | Global settings | +| `conf.d/metrics.conf` | Metrics service settings | +| `logback.xml` | Log level and output configuration | +| `ca/` | CA certificates, CRL, signed certs, and CSRs (`cadir`) | + +**Puppet configuration** (`/etc/puppetlabs/puppet/`): + +| File | Description | +| ---- | ----------- | +| `puppet.conf` | Main Puppet configuration — server settings go in the `[server]` section | +| `ssl/` | Server node certificate, private key, and CRL (`ssldir`) | +| `hiera.yaml` | Hiera lookup configuration | + +## Code directory + +Puppet manifests and modules live under `/etc/puppetlabs/code/`. This is the +[codedir](/openvox/latest/dirs_codedir.html) used by the server when compiling catalogs. + +## Runtime data + +| Path | Description | +| ---- | ----------- | +| `/opt/puppetlabs/server/data/puppetserver/` | Server runtime data (reports, state, JRuby gems, yaml cache) | +| `/var/run/puppetlabs/puppetserver/` | PID file | + +## Log files + +| Path | Description | +| ---- | ----------- | +| `/var/log/puppetlabs/puppetserver/puppetserver.log` | Main server log | +| `/var/log/puppetlabs/puppetserver/puppetserver-access.log` | HTTP access log | +| `/var/log/puppetlabs/puppetserver/puppetserver-status.log` | Status service log | + +## Service + +OpenVox Server runs as the `puppetserver` service, managed by systemd. + +| Property | Value | +| -------- | ----- | +| Service name | `puppetserver` | +| Runs as user | `puppet` (created by the package installer) | +| JVM heap config | `/etc/sysconfig/puppetserver` (EL) or `/etc/default/puppetserver` (Debian/Ubuntu) | + +## Migrating from Puppet Server packages + +OpenVox Server uses the same directory layout as Puppet Server. If you are replacing +legacy Puppet Server packages: + +- Back up `/etc/puppetlabs/` and `/opt/puppetlabs/server/data/` before you begin. +- Expect the same command names, configuration paths, and service name after installation. +- Do not install Puppet Server and OpenVox Server packages side by side on the same host. diff --git a/docs/_openvox-server_9x/intermediate_ca.md b/docs/_openvox-server_9x/intermediate_ca.md new file mode 100644 index 000000000..372f856ec --- /dev/null +++ b/docs/_openvox-server_9x/intermediate_ca.md @@ -0,0 +1,96 @@ +--- +layout: default +title: "OpenVox Server: Intermediate CA" +--- + +Before starting OpenVox Server for the first time, you need to initialize the CA. How you do +that depends on whether you want OpenVox to manage your CA or whether you have an existing +external CA. + +**OpenVox manages your CA:** +Run `puppetserver ca setup` to generate an intermediate CA with all necessary certificates and keys. + +The following diagram shows the configuration of OpenVox’s basic certificate infrastructure. + +![A diagram showing OpenVox’s basic certificate infrastructure](ca_basic_foss.png) + +**You have an external certificate authority:** +An external certificate authority is your organization's own PKI infrastructure, managed +independently of OpenVox. Create a cert chain from it and use `puppetserver ca import` to +install the chain on your server. +OpenVox agents handle an intermediate CA out of the box — no need to copy files around by hand +or configure CRL checking. + +If you skip both commands and start OpenVox Server directly, the server will generate a +non-intermediate CA for backward compatibility. This configuration is not recommended for +new deployments. + +## Where to set CA configuration + +All CA configuration takes place in OpenVox’s config file. See the [OpenVox Configuration Reference](/openvox/latest/configuration.html) for details. + +## Set up OpenVox as an intermediate CA with an external root + +OpenVox Server needs to present the full certificate chain to clients so the client can authenticate the +server. You construct the certificate chain by concatenating the CA certificates within a PEM file, starting with +the new intermediate CA certificate and descending to the root CA certificate. + +The following diagram shows the configuration of OpenVox’s certificate infrastructure with an external root. + +![A diagram showing OpenVox’s certificate infrastructure with an external root](ca_external_root_foss.png) + +To set up OpenVox as an intermediate CA with an external root: + +1. Collect your organization’s chain of trust. This includes: + - The root cert + - Any intermediate CA certs + - The CA cert that you will use to issue a CA signing cert for your OpenVox infrastructure +1. Collect the corresponding CRLs for each of these certificates. Take note of the expiration dates of each + CRL. When one expires, you need to refresh it for OpenVox to continue working. +1. Create a private RSA key (minimum 2048-bit) with no passphrase for the OpenVox CA — you will need to + import it into your OpenVox infrastructure later. `puppetserver ca import` requires the key to be + unencrypted. +1. Create a CSR for the OpenVox CA and sign it with SHA-256 using the appropriate cert from your + organization’s trust chain, which you gathered in Step 1. This is the new OpenVox CA cert, which will + be used to sign all other OpenVox infrastructure certs. The signed cert must have the following + extensions set: + - `basicConstraints: CA:TRUE` (critical) + - `keyUsage: keyCertSign, cRLSign` (critical) + - `subjectKeyIdentifier: hash` + - `authorityKeyIdentifier: keyid:always` +1. Create a CRL for the new OpenVox CA cert. +1. Concatenate all of the certs into a PEM file, starting with the new OpenVox CA cert and ending with your + organization’s root cert. The file should contain the PEM-encoded certs, like this: + + ```text + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + ``` + +1. Concatenate all of the CRLs into a PEM file, in the same order as the certificates. The file should + contain the PEM-encoded CRLs, like this: + + ```text + -----BEGIN X509 CRL----- + + -----END X509 CRL----- + -----BEGIN X509 CRL----- + + -----END X509 CRL----- + -----BEGIN X509 CRL----- + + -----END X509 CRL----- + ``` + +1. Use the `puppetserver ca import` command to trigger the rest of the CA setup: + + ```bash + puppetserver ca import --cert-bundle ca-bundle.pem --crl-chain crls.pem --private-key openvox_ca_key.pem + ``` diff --git a/docs/_openvox-server_9x/known_issues.markdown b/docs/_openvox-server_9x/known_issues.markdown new file mode 100644 index 000000000..bfe48f497 --- /dev/null +++ b/docs/_openvox-server_9x/known_issues.markdown @@ -0,0 +1,46 @@ +--- +layout: default +title: "OpenVox Server: Known Issues" +--- + +For a list of all known issues, visit the [OpenVox Server issue tracker](https://github.com/OpenVoxProject/openvox-server/issues). + +## Potential JAVA ARGS settings + +If you're working outside of lab environment, increase `ReservedCodeCache` to `512m` under normal load. If you're working with 6-12 JRuby instances (or a `max-requests-per-instance` value significantly less +than 100k), run with a `ReservedCodeCache` of 1G. Twelve or more JRuby instances in a single server might require 2G or more. + +Similar caveats regarding scaling `ReservedCodeCache` might apply if users are managing `MaxMetaspace`. + +## `tmp` directory mounted `noexec` + +In some cases (especially for RHEL installations) if the `/tmp` directory is mounted as `noexec`, OpenVox Server may fail to run correctly, and you may see an error in the OpenVox Server logs similar to the +following: + +```text +Nov 12 17:46:12 fqdn.com java[56495]: Failed to load feature test for posix: can't find user for 0 +Nov 12 17:46:12 fqdn.com java[56495]: Cannot run on Microsoft Windows without the win32-process, win32-dir and win32-service gems: Win32API only supported on win32 +Nov 12 17:46:12 fqdn.com java[56495]: Puppet::Error: Cannot determine basic system flavour +``` + +This is caused by the fact that JRuby contains some embedded files which need to be copied somewhere on the filesystem before they can be executed +([see this JRuby issue](https://github.com/jruby/jruby/issues/2186)). To work around this issue, you can either mount the `/tmp` directory without `noexec`, or you can choose a different directory to use as the +temporary directory for the OpenVox Server process. + +Either way, you'll need to set the permissions of the directory to `1777`. This allows the OpenVox Server JRuby process to write a file to `/tmp` and then execute it. If permissions are set incorrectly, you'll +get a massive stack trace without much useful information in it. + +To use a different temporary directory, you can set the following JVM property: + +```text +-Djava.io.tmpdir=/some/other/temporary/directory +``` + +When OpenVox Server is installed from packages, add this property to the `JAVA_ARGS` and `JAVA_ARGS_CLI` variables defined in either `/etc/sysconfig/puppetserver` or `/etc/default/puppetserver`, depending on +your distribution. Invocations of the `gem`, `ruby`, and `irb` subcommands use the updated `JAVA_ARGS_CLI` on their next invocation. The service will need to be restarted in order to re-read the `JAVA_ARGS` +variable. + +## OpenVox Server fails to connect to load-balanced servers with different SSL certificates + +Intermittent SSL connection failures have been seen when OpenVox Server tries to make SSL requests to servers via the same virtual IP address where the servers present different certificates during the SSL +handshake. For more information, see [this page](./ssl_server_certificate_change_and_virtual_ips.html). diff --git a/docs/_openvox-server_9x/metrics-api/v1/metrics_api.markdown b/docs/_openvox-server_9x/metrics-api/v1/metrics_api.markdown new file mode 100644 index 000000000..9a70fecde --- /dev/null +++ b/docs/_openvox-server_9x/metrics-api/v1/metrics_api.markdown @@ -0,0 +1,81 @@ +--- +layout: default +title: "OpenVox Server: Metrics API v1" +--- + +By default, OpenVox Server enables two optional web APIs for +[Java Management Extension (JMX)](https://docs.oracle.com/javase/tutorial/jmx/index.html) +metrics, namely [managed beans (MBeans)](https://docs.oracle.com/javase/tutorial/jmx/mbeans/). For the newer Jolokia-based metrics API, see [the `/metrics/v2` documentation](../v2/metrics_api.html). + +The metrics v1 API was introduced in Puppet Enterprise 2016.4 and was open sourced afterwards. It is still enabled but is deprecated. + +> **Note:** The metrics described here are returned only when passing the `level=debug` URL parameter, and the structure of the returned data might change, or the endpoint might be removed, in future versions. + +## `GET /metrics/v1/mbeans` + +The `GET /metrics/v1/mbeans` endpoint lists available MBeans. + +### Response keys + +- The key is the name of a valid MBean. +- The value is a URI to use when requesting that MBean's attributes. + +## `POST /metrics/v1/mbeans` + +The `POST /metrics/v1/mbeans` endpoint retrieves requested MBean metrics. + +### Query parameters + +The query doesn't require any parameters, but the request body must contain a JSON object whose values are metric names, or a JSON array of metric names, or a JSON string containing a single metric's name. + +For a list of metric names, make a `GET` request to `/metrics/v1/mbeans`. + +### Response keys + +The response format, though always JSON, depends on the request format: + +- Requests with a JSON object return a JSON object where the values of the original object are transformed into the Mbeans' attributes for the metric names. +- Requests with a JSON array return a JSON array where the items of the original array are transformed into the Mbeans' attributes for the metric names. +- Requests with a JSON string return a JSON object of the Mbean's attributes for the given metric name. + +## GET /metrics/v1/mbeans/\ + +The `GET /metrics/v1/mbeans/` endpoint reports on a single metric. + +### Query parameters + +The query doesn't require any parameters, but the endpoint itself must correspond to one of the metrics returned by a `GET` request to `/metrics/v1/mbeans`. + +### Response keys + +The endpoint's responses contain a JSON object mapping strings to values. The keys and values returned in the response vary based on the specified metric. + +### Example + +Use `curl` from localhost to request data on MBean memory usage: + +```console +curl 'http://localhost:8080/metrics/v1/mbeans/java.lang:type=Memory' +``` + +The response should contain a JSON object representing the data: + +```json +{ + "ObjectPendingFinalizationCount" : 0, + "HeapMemoryUsage" : { + "committed" : 807403520, + "init" : 268435456, + "max" : 3817865216, + "used" : 129257096 + }, + "NonHeapMemoryUsage" : { + "committed" : 85590016, + "init" : 24576000, + "max" : 184549376, + "used" : 85364904 + }, + "Verbose" : false, + "ObjectName" : "java.lang:type=Memory" +} +``` diff --git a/docs/_openvox-server_9x/metrics-api/v2/metrics_api.markdown b/docs/_openvox-server_9x/metrics-api/v2/metrics_api.markdown new file mode 100644 index 000000000..6a7c460ba --- /dev/null +++ b/docs/_openvox-server_9x/metrics-api/v2/metrics_api.markdown @@ -0,0 +1,180 @@ +--- +layout: default +title: "OpenVox Server: Metrics API v2" +--- + +By default, OpenVox Server enables two optional web APIs for [Java Management Extension (JMX)](https://docs.oracle.com/javase/tutorial/jmx/index.html) metrics, namely [managed beans (MBeans)](https://docs.oracle.com/javase/tutorial/jmx/mbeans/). +For the older metrics API, see [the `/metrics/v1` documentation](../v1/metrics_api.html). + +## Jolokia endpoints + +The v2 metrics endpoint uses the [Jolokia](https://jolokia.org) library, an extensive open-source metrics library with its own documentation. + +The documentation below provides only the information you need to +use the metrics as configured by default for OpenVox Server, but Jolokia offers more features than are described below. Consult the +[Jolokia documentation](https://jolokia.org/documentation.html) for more information. + +For security reasons, we enable only the read-access Jolokia interface by default: + +- `read` +- `list` +- `version` +- `search` + +### Configuring Jolokia + +To change the security access policy, create the `/etc/puppetlabs/puppetserver/jolokia-access.xml` file with contents that follow the [Jolokia access policy](https://jolokia.org/reference/html/security.html) +and uncomment the `metrics.metrics-webservice.jolokia.servlet-init-params.policyLocation` parameter before restarting OpenVox Server. + +The `metrics.metrics-webservice.jolokia.servlet-init-params` table within the [`/etc/puppetlabs/puppetserver/conf.d/metrics.conf`](../../config_file_metrics.html) file provides more configuration options. +See Jolokia's [agent initialization documentation](https://jolokia.org/reference/html/agents.html#agent-war-init-params) for all of the available options. + +### Disabling the endpoints + +To disable the v2 endpoints, set the `metrics.metrics-webservice.jolokia.enabled` parameter in `metrics.conf` to `false`. + +## Usage + +You can query the metrics v2 API using `GET` or `POST` requests. + +### `GET /metrics/v2/` + +(Introduced in Puppet Server 5) + +This endpoint requires an operation, and depending on the operation can accept or might require an additional query: + +```http +GET /metrics/v2// +``` + +#### Response + +A successful request returns a JSON document. + +#### Examples + +To list all valid mbeans querying the metrics endpoint + +```http +GET /metrics/v2/list +``` + +Which should return a response similar to + +```json +{ + "request": { + "type": "list" + }, + "value": { + "java.util.logging": { + "type=Logging": { + "op": { + "getLoggerLevel": { + ... + }, + ... + }, + "attr": { + "LoggerNames": { + "rw": false, + "type": "[Ljava.lang.String;", + "desc": "LoggerNames" + }, + "ObjectName": { + "rw": false, + "type": "javax.management.ObjectName", + "desc": "ObjectName" + } + }, + "desc": "Information on the management interface of the MBean" + } + }, + ... + } +} +``` + +So, from the example above we could query for the registered logger names with this HTTP call: + +```http +GET /metrics/v2/read/java.util.logging:type=Logging/LoggerNames +``` + +Which would return the JSON document + +```json +{ + "request": { + "mbean": "java.util.logging:type=Logging", + "attribute": "LoggerNames", + "type": "read" + }, + "value": [ + "javax.management.snmp", + "global", + "javax.management.notification", + "javax.management.modelmbean", + "javax.management.timer", + "javax.management", + "javax.management.mlet", + "javax.management.mbeanserver", + "javax.management.snmp.daemon", + "javax.management.relation", + "javax.management.monitor", + "javax.management.misc", + "" + ], + "timestamp": 1497977258, + "status": 200 +} +``` + +The MBean names can then be created by joining the the first two keys of the value table with a colon (the `domain` and `prop list` in Jolokia parlance). Querying the MBeans is achieved via the `read` +operation. The `read` operation has as its GET signature: + +```http +GET /metrics/v2/read/// +``` + +### `POST /metrics/v2/` + +You can also submit a POST request with the query as a JSON document in the body of the POST. + +## Filtering + +The new Jolokia-based metrics API also provides globbing (wildcard selection) and response filtering features. + +### Example + +You can combine both of these features to query garbage collection data, but return only the collection counts and times. + +```text +GET metrics/v2/read/java.lang:name=*,type=GarbageCollector/CollectionCount,CollectionTime +``` + +This returns a JSON response: + +```json +{ + "request": { + "mbean": "java.lang:name=*,type=GarbageCollector", + "attribute": ["CollectionCount", "CollectionTime"], + "type": "read" + }, + "value": { + "java.lang:name=PS Scavenge,type=GarbageCollector": { + "CollectionTime": 1314, + "CollectionCount": 27 + }, + "java.lang:name=PS MarkSweep,type=GarbageCollector": { + "CollectionTime": 580, + "CollectionCount": 5 + } + }, + "timestamp": 1497977710, + "status": 200 +} +``` + +Refer to the [Jolokia protocol documentation](https://jolokia.org/reference/html/protocol.html) for more advanced usage. diff --git a/docs/_openvox-server_9x/pson.md b/docs/_openvox-server_9x/pson.md new file mode 100644 index 000000000..5b6b48628 --- /dev/null +++ b/docs/_openvox-server_9x/pson.md @@ -0,0 +1,62 @@ +--- +layout: default +title: "OpenVox Server HTTP API: PSON" +--- + +# PSON + +> **Deprecated:** PSON support is retained for backward compatibility but should not be used in new integrations. +> All OpenVox HTTP API endpoints that previously supported PSON now prefer JSON (`application/json`). PSON remains +> an accepted fallback format for binary content that cannot be represented as valid UTF-8. + +PSON is a variant of [JSON](http://json.org) that OpenVox uses for serializing data to transmit across the network +or store on disk. Whereas JSON requires that the serialized form is valid unicode (usually UTF-8), PSON is 8-bit +ASCII, which allows it to represent arbitrary byte sequences in strings. + +OpenVox uses the MIME types `"pson"` and `"text/pson"` to refer to PSON. + +## Differences from JSON + +PSON does *not differ* from JSON in its representation of objects, arrays, numbers, booleans, and null values. +PSON *does* serialize strings differently from JSON. + +A PSON string is a sequence of 8-bit ASCII encoded data. It must start and end with `"` (ASCII 0x22) characters. +Between these characters it may contain any byte sequence. Some individual characters are represented by a sequence +of characters: + +| Byte | ASCII Character | Encoded Sequence | Encoded ASCII Sequence | +| ---- | --------------- | ---------------- | ---------------------- | +| 0x22 | " | 0x5C, 0x22 | \" | +| 0x5c | \ | 0x5C, 0x5C | \\ | +| 0x08 | Backspace | 0x5C, 0x62 | \b | +| 0x09 | Horizontal Tab | 0x5C, 0x74 | \t | +| 0x0A | Line Feed | 0x5C, 0x6E | \n | +| 0x0C | Form Feed | 0x5C, 0x66 | \f | +| 0x0D | Carriage Return | 0x5C, 0x72 | \r | + +In addition, any character between 0x00 and 0x1F (except the ones listed above) must be encoded as a six byte +sequence of `\u` followed by four ASCII digits of the hex number of the desired character. For example the ASCII +Record Separator character (0x1E) is represented as `` (0x5C, 0x75, 0x30, 0x30, 0x31, 0x45). + +## Decoding PSON Using JSON Parsers + +Many languages have JSON parsers already, which can often be used to parse PSON data. Although JSON requires that +it is encoded as unicode, most parsers will produce usable output from PSON if they are instructed to interpret the +input as Latin-1 encoding. + +In all these examples there is a file available called `data.pson` that contains the ruby structure +`{ "data" => "\x07\x08\xC3\xC3" }` encoded as PSON (the value is an invalid unicode sequence). In bytes the data +is: + + 0x7b 0x22 0x64 0x61 0x74 0x61 0x22 0x3a 0x22 0x5c 0x75 0x30 0x30 0x30 0x37 0x5c 0x62 0xc3 0xc3 0x22 0x7d + +Python Example: + + >>> import json + >>> json.load(open("data.pson"), "latin_1") + {u'data': u'\x07\x08\xc3\xc3'} + +Clojure Example: + + user> (parse-string (slurp "data.pson" :encoding "ISO-8859-1")) + {"data" "^G\bÃÃ"} diff --git a/docs/_openvox-server_9x/puppet-api/v3/compile.markdown b/docs/_openvox-server_9x/puppet-api/v3/compile.markdown new file mode 100644 index 000000000..900c2dfb4 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/compile.markdown @@ -0,0 +1,88 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Compile" +--- + +[PCore Generic Data]: https://github.com/OpenVoxProject/puppet-specifications/blob/master/language/data-types/pcore-generic-data.md +[`auth.conf` documentation]: ../../config_file_auth.html +[v4 catalog schema]: ../v4/catalog.json + +The compile endpoint allows compilation of arbitrary pcore serialized Puppet Code ASTs (aka parsed, but unevaluated Puppet Code snippets). +In addition to the code AST, the caller must provide the `certname`, `environment`, `facts`, `trusted_facts`, and `variables` for compilation (the context in which this +AST will be evaluated). +Optionally, the caller may provide `job_id`, +`transaction_id`, and configure returned logging. The request body must be JSON formatted and the caller must accept a JSON response. The server must not have `rich_data` disabled. + +## `POST /puppet/v3/compile` + +The request body must look like: + +```json +{ + "code_ast": "", + "certname": "", + "environment": "", + "facts": { "values": { "": , ... } }, + "trusted_facts": { "values": { "": , ... } }, + "variables": { "values": { "": , ... } }, + + # The rest are optional: + "transaction_uuid": "", + "job_id": "", + "options": { "capture_logs": , + "log_level": + "compile_for_plan": } +} +``` + +### `code_ast` (required) + +A parsed string of json encoded [PCore Generic Data][] +objects representing a Puppet Code AST. When `rich_data` is enabled, the AST represents an intermediate step when compiling Puppet Code to a catalog. +The returned catalog is equivalent to a catalog returned with the MIME type `application/vnd.puppet.rich+json` normally. + +### `certname` (required) + +The name of the node for which the AST will be compiled. + +### `environment` (required) + +The name of the environment within which to compile the AST. + +### `facts` (required) + +A hash with a required `values` key, containing a hash of all the facts for the node. + +### `trusted_facts` (required) + +A hash with a required `values` key containing a hash of the trusted facts for a node. +In a normal agent's catalog request, these would be extracted from the cert, but this endpoint does not require a cert for the node whose catalog is being compiled. + +### `variables` (required) + +A hash with a required `values` key, containing a hash of all the variables in scope for the compilation. + +### `transaction_uuid` + +The id for tracking the AST compilation. + +### `job_id` + +The id of the orchestrator job that requested this compilation. + +### `options` + +A hash of options beyond direct input to compilation. + +`capture_logs`: Whether to return the logging events that occurred during compilation. + +`log_level` :A string representing the logging level at which log events should be captured. Valid values are one of "debug", "warn", "info". + +## Schema + +The response body conforms to the [v4 catalog schema][]. + +## Authorization + +All requests made to the compile API are authorized using the Trapperkeeper-based `auth.conf`. +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][]. diff --git a/docs/_openvox-server_9x/puppet-api/v3/environment_classes.json b/docs/_openvox-server_9x/puppet-api/v3/environment_classes.json new file mode 100644 index 000000000..ae3e6f57f --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/environment_classes.json @@ -0,0 +1,76 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Environment Classes", + "description": "Information about the classes in a Puppet code environment", + "type": "object", + "properties": { + "files": { + "description": "The array of manifest files which exist in an environment. Even files that do not contain any classes will appear in the array.", + "type": "array", + "items": { + "type": "object", + "properties": { + "path": { + "description": "Fully-qualified path on the server to the manifest", + "type": "string" + }, + "classes": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "description": "Name of the class. For example, if the class were defined as 'class myclass', the name would be 'myclass'.", + "type": "string" + }, + "params": { + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "description": "Name of the class parameter. For example, if a class parameter were defined as 'String $my_param', the name would be 'my_param'.", + "type": "string" + }, + "type": { + "description": "Data type, if defined, for the class parameter. For example, if a class parameter were defined as 'String $my_param', the data type would be 'String'. See https://doc.openvoxproject.org/openvox/latest/lang_data.html for information on the available data types.", + "type": "string" + }, + "default_source": { + "description": "Source text for the default value, if defined, for the class parameter. For example, if a class parameter were defined in the manifest as 'String $my_param = \"my value\"', the default_source would be '\"my value\"' where \" would appear literally in the text. Note that no expressions in the default value - for example, references to other variables - are expanded. The text for default_source matches the exact content in the parsed manifest.", + "type": "string" + }, + "default_literal": { + "description": "Literal representation of the default value, if defined, for the class parameter. This member is only present if the value can be expressed using primitive JSON data types. For example, if a class parameter were defined in the manifest as 'Integer $my_integer = 3', the default_literal would be a JSON number whereas the corresponding default_source would be a JSON string containing '3'. The default_literal would be omitted for the following cases: 1) The primitive value has no direct translation into a JSON primitive type (for example, a regular expression, default, or undef). 2) The value contains a hash - either top-level or nested under another array or hash - with at least one key that is not a string. 3) The value contains one or more expressions that would have to be evaluated at catalog compilation time in order for the actual default value to be determined. See https://docs.openvoxproject.org/openvox/latest/lang_data.html for information on the available data types and values." + } + }, + "required": ["name"], + "additionalProperties": false + } + } + }, + "required": ["name", "params"], + "additionalProperties": false + } + }, + "error": { + "description": "If an error was encountered during manifest parsing, this member will be present instead of 'classes'. The error string provides details about the specific error.", + "type": "string" + } + }, + "required": ["path"], + "oneOf": [ + {"required": ["classes"]}, + {"required": ["error"]} + ], + "additionalProperties": false + } + }, + "name": { + "description": "Name of the environment", + "type": "string" + } + }, + "required": ["files", "name"], + "additionalProperties": false +} diff --git a/docs/_openvox-server_9x/puppet-api/v3/environment_classes.markdown b/docs/_openvox-server_9x/puppet-api/v3/environment_classes.markdown new file mode 100644 index 000000000..ada419d3d --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/environment_classes.markdown @@ -0,0 +1,321 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Environment Classes" +--- + +[classes]: /openvox/latest/lang_classes.html +[node definitions]: /openvox/latest/lang_node_definitions.html +[defined types]: /openvox/latest/lang_defined_types.html +[`environment_timeout`]: /openvox/latest/config_file_environment.html#environment_timeout +[`manifest` setting]: /openvox/latest/config_file_environment.html#manifest +[`auth.conf`]: ../../config_file_auth.html +[environment cache API]: ../../admin-api/v1/environment-cache.html +[Etag]: https://tools.ietf.org/html/rfc7232#section-2.3 + +The environment classes API will return information about what classes are installed for the requested environment. + +## Changes in the environment classes API + +Compared to the resource type API, the environment classes API covers different things, returns new or different information, and omits some information. + +### Covers classes only + +The environment classes API covers only [classes][], whereas the resource type API covers classes, [node definitions][], and [defined types][]. + +### Changes class information caching behavior + +Queries to the resource type API use cached class information per the configuration of the [`environment_timeout`][] setting, as set in the corresponding environment's `environment.conf` file. +The environment classes API does not use the value of `environment_timeout` with respect to the data that it caches. +Instead, only when the `environment-class-cache-enabled` setting in the `jruby-puppet` configuration section is set to `true`, +the environment classes API uses HTTP [Etags][Etag] to represent specific versions of the class information. +And it uses the OpenVox Server [environment cache API][] as an explicit mechanism for marking an Etag as expired. +See the [Headers and caching behavior](#headers-and-caching-behavior) section for more information about caching and invalidation of entries. + +### Uses typed values + +The environment classes API includes a `type`, if defined for a class parameter. For example, if the class parameter were defined as `String $some_str`, the `type` parameter would hold a value of `String`. + +### Provides default literal values + +For values that can be presented in pure JSON, the environment classes API provides a `default_literal` form of a class parameter's default value. +For example, if an `Integer` type class parameter were defined +in the manifest as having a default value of `3`, the `default_literal` element for the parameter will contain a JSON Number type of 3. + +### Lacks filters + +The environment classes API does not provide a way to filter the list of classes returned via use of a search string. +The environment classes API returns information for all classes found within an environment's manifest files. + +### Includes filenames + +The resource type API includes the filename in which each class was found. + +### Lacks line numbers + +The environment classes API does not include the line number at which a class is found in the file. + +### Lacks documentation strings + +The environment classes API does not include any doc strings for a class entry. + +### Returns file entries for manifests with no classes + +The environment classes API returns a file entry for manifests that exist in the environment but in which no classes were found. The resource type API omits entries for files which do not contain any classes. + +### Uses `application/json` Content-Type + +The Content-Type in the response to an environment classes API query is `application/json`, whereas the resource type API uses a Content-Type of `text/pson`. + +### Includes successfully parsed classes, even if some return errors, and returns error messages + +The environment classes API includes information for every class that can successfully be parsed. +For any errors which occur when parsing individual manifest files, the response includes an entry for the corresponding manifest file, along with an error and detail string about the failure. + +In comparison, if an error is encountered when parsing a manifest, the resource type API omits information from the manifest entirely. +It includes class information from other manifests that it successfully parsed, assuming none of the parsing errors were found in one of the files associated with the environment's [`manifest` setting][]. +If one or more classes is returned but errors were encountered parsing other manifests, the response from the resource type API call doesn't include any explicit indication that a parsing error was encountered. + +## `GET /puppet/v3/environment_classes?environment=:environment` + +Making a request with no query parameters is not supported and returns an HTTP 400 (Bad Request) response. + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Query Parameters + +Provide one parameter to the GET request: + +- `environment`: Only the classes and parameter information pertaining to the specified environment will be returned for the call. + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/environment_classes?environment=env + +HTTP/1.1 200 OK +Etag: b02ede6ecc432b134217a1cc681c406288ef9224 +Content-Type: application/json + +{ + "files": [ + { + "path": "/etc/puppetlabs/code/environments/env/manifests/site.pp", + "classes": [] + }, + { + "path": "/etc/puppetlabs/code/environments/env/modules/mymodule/manifests/init.pp", + "classes": [ + { + "name": "mymodule", + "params": [ + { + "default_literal": "this is a string", + "default_source": "\"this is a string\"", + "name": "a_string", + "type": "String" + }, + { + "default_literal": 3, + "default_source": "3", + "name": "an_integer", + "type": "Integer" + } + ] + } + ] + }, + { + "error": "Syntax error at '=>' at /etc/puppetlabs/code/environments/env/modules/mymodule/manifests/other.pp:20:19", + "path": "/etc/puppetlabs/code/environments/env/modules/mymodule/manifests/other.pp" + } + ], + "name": "env" +} +``` + +#### GET request with Etag roundtripped from a previous GET request + +If you send the [Etag][] value that was returned from the previous request to the server in a follow-up request, and the underlying environment cache has not been invalidated, +the server will return an HTTP 304 (Not Modified) response. +See the [Headers and Caching Behavior](#headers-and-caching-behavior) section for more information about caching and invalidation of entries. + +```text +GET /puppet/v3/environment_classes?environment=env +If-None-Match: b02ede6ecc432b134217a1cc681c406288ef9224 + +HTTP/1.1 304 Not Modified +Etag: b02ede6ecc432b134217a1cc681c406288ef9224 +``` + +If the environment cache has been updated from what was used to calculate the original Etag, the server will return a response with the full set of environment class information: + +```text +GET /puppet/v3/environment_classes?environment=env +If-None-Match: b02ede6ecc432b134217a1cc681c406288ef9224 + +HTTP/1.1 200 OK +Etag: 2f4f83096265b9741c5304b3055f866df0336762 +Content-Type: application/json + +{ + "files": [ + { + "path": "/etc/puppetlabs/code/environments/env/manifests/site.pp", + "classes": [] + }, + { + "path": "/etc/puppetlabs/code/environments/env/modules/mymodule/manifests/init.pp", + "classes": [ + { + "name": "mymodule", + "params": [ + { + "default_literal": "this is a string", + "default_source": "\"this is a string\"", + "name": "a_string", + "type": "String" + }, + { + "default_literal": 3, + "default_source": "3", + "name": "an_integer", + "type": "Integer" + }, + { + "default_literal": { + "one": "foo", + "two": "hello" + }, + "default_source": "{ \"one\" => \"foo\", \"two\" => \"hello\" }", + "name": "a_hash", + "type": "Hash" + } + ] + } + ] + } + ], + "name": "env" +} +``` + +#### Environment does not exist + +If you send a request with an environment parameter that doesn't correspond to the name of a directory environment on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/environment_classes?environment=doesnotexist + +HTTP/1.1 404 Not Found + +Could not find environment 'doesnotexist' +``` + +#### No environment given + +```text +GET /puppet/v3/environment_classes + +HTTP/1.1 400 Bad Request + +You must specify an environment parameter. +``` + +#### Environment parameter specified with no value + +```text +GET /puppet/v3/environment_classes?environment= + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not '' +``` + +#### Environment includes non-alphanumeric characters + +If the environment parameter in your request includes any characters that are not `A-Z`, `a-z`, `0-9`, or `_` (underscore), the server returns an HTTP 400 (Bad Request) error: + +```text +GET /puppet/v3/environment_classes?environment=bog|us + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not 'bog|us' +``` + +### Schema + +An environment classes response body conforms to the [environment classes schema](./environment_classes.json). + +### Headers and Caching Behavior + +If the `environment-class-cache-enabled` setting in the `jruby-puppet` configuration section is set to `true`, the environment classes API caches the response data. +This can provide a significant performance benefit by reducing the amount of data that needs to be provided in a response when the underlying Puppet code on disk remains unchanged from one request to the next. +Use of the cache does, however, require that cache entries are invalidated after Puppet code has been updated. + +To avoid invalidated cache entries, you can omit the `environment-class-cache-enabled` setting from a node's configuration or set it to `false`. +In this case, the server discovers and parses manifests for every incoming request. +This can significantly increase bandwidth overhead for repeated requests, particularly when there are few changes to the underlying Puppet code. +However, this approach ensures that the latest available data is returned to every request. + +#### Behaviors when the environment class cache is enabled + +When the `environment-class-cache-enabled` setting is set to `true`, the response to a query to the `environment_classes` endpoint includes an HTTP [Etag][] header. +The value for the Etag header is a hash that represents the state of the latest class information available for the requested environment. For example: + +```text +ETag: 31d64b8038258202b4f5eb508d7dab79c46327bb +``` + +A client can (but is not required to) provide the Etag value back to the server in a subsequent `environment_classes` request. The client would provide the tag value as the value for an +[If-None-Match](https://tools.ietf.org/html/rfc7232#section-3.2) HTTP header: + +```text +If-None-Match: 31d64b8038258202b4f5eb508d7dab79c46327bb +``` + +If the latest state of code available on the server matches that of the value in the `If-None-Match` header, the server returns an HTTP 304 (Not Modified) response with no response body. +If the server has newer code available than what is captured by the `If-None-Match` header value, or if no `If-None-Match` header is provided in the request, the server parses manifests again. +Assuming the resulting payload is different than a previous request's, the server provides a different Etag value and new class information in the response payload. + +If the client sends an `Accept-Encoding: gzip` HTTP header for the request and the server provides a gzip-encoded response body, the server might append the characters `--gzip` to the end of the Etag. +For example, the HTTP response headers could include: + +```text +Content-Encoding: gzip +ETag: e84bbce5482243b3eb3a190e5c90e535cf4f20de--gzip +``` + +The server accepts both forms of an Etag (with or without the trailing `--gzip` characters) as the same value when validating it in a request's `If-None-Match` header against its cache. + +It is best, however, for clients to use the Etag without parsing its content. +A client expecting an HTTP 304 (Not Modified) response if the cache has not been updated since the prior request should provide the exact value returned in the `Etag` header from one request, +to the server in an `If-None-Match` header in a subsequent request for the environment's class information. + +#### Clearing class information cache entries + +After updating an environment's manifests, you must clear the server's class information cache entries, so the server can parse the latest manifests and reflect class changes to the class information in queries +to the environment classes endpoint. To clear cache entries on the server, do one of the following: + +- Call the [`environment-cache` API endpoint][environment cache API]. + + For best performance, call this endpoint with a query parameter that specifies the environment whose cache should be flushed. + +- Restart OpenVox Server. + + Each environment's cache is held in memory for the OpenVox Server process and is effectively flushed whenever OpenVox Server is restarted, whether with a [HUP signal, service reload,](../../restarting.html) or + a full JVM restart. + +### Authorization + +All requests made to the environment classes API are authorized using the Trapperkeeper-based [`auth.conf`][]. + +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][`auth.conf`]. diff --git a/docs/_openvox-server_9x/puppet-api/v3/environment_modules.json b/docs/_openvox-server_9x/puppet-api/v3/environment_modules.json new file mode 100644 index 000000000..fcfee39f6 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/environment_modules.json @@ -0,0 +1,33 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Environment Modules", + "description": "Information about the modules in a Puppet code environment", + "type": "object", + "properties": { + "modules": { + "description": "The array of modules which exist in an environment.", + "type": "array", + "items": { + "type": "object", + "properties": { + "name": { + "description": "The name of the puppet module", + "type": "string" + }, + "version": { + "description": "The version of the puppet module", + "type": ["string","null"] + } + }, + "required": ["name", "version"], + "additionalProperties": false + } + }, + "name": { + "description": "Name of the environment", + "type": "string" + } + }, + "required": ["modules", "name"], + "additionalProperties": false +} diff --git a/docs/_openvox-server_9x/puppet-api/v3/environment_modules.markdown b/docs/_openvox-server_9x/puppet-api/v3/environment_modules.markdown new file mode 100644 index 000000000..6b0e1d5ab --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/environment_modules.markdown @@ -0,0 +1,204 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Environment Modules" +--- + +[`auth.conf`]: ../../config_file_auth.html + +The environment modules API will return information about what modules are installed for the requested environment. + +## `GET /puppet/v3/environment_modules` + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/environment_modules + +HTTP/1.1 200 OK +Content-Type: application/json + +[{ + "modules": [ + { + "name": "puppet/systemd", + "version": "9.4.0" + }, + { + "name": "puppetlabs/stdlib", + "version": "9.7.0" + } + ], + "name": "env" +}, +{ + "modules": [ + { + "name": "puppetlabs/stdlib", + "version": "9.6.0" + }, + { + "name": "puppet/systemd", + "version": "9.2.0" + } + ], + "name": "production" +}] +``` + +## `GET /puppet/v3/environment_modules?environment=:environment` + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Query Parameters + +Provide one parameter to the GET request: + +* `environment`: Request information about modules pertaining to the specified environment only. + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/environment_modules?environment=env + +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "modules": [ + { + "name": "puppet/systemd", + "version": "9.4.0" + }, + { + "name": "puppetlabs/stdlib", + "version": "9.7.0" + } + ], + "name": "env" +} +``` + +#### Environment does not exist + +If you send a request with an environment parameter that doesn't correspond to the name of a directory environment on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/environment_modules?environment=doesnotexist + +HTTP/1.1 404 Not Found + +Could not find environment 'doesnotexist' +``` + +#### No environment given + +```text +GET /puppet/v3/environment_modules + +HTTP/1.1 400 Bad Request + +An environment parameter must be specified +``` + +#### Environment parameter specified with no value + +```text +GET /puppet/v3/environment_modules?environment= + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not '' +``` + +#### Environment includes non-alphanumeric characters + +If the environment parameter in your request includes any characters that are not `A-Z`, `a-z`, `0-9`, or `_` (underscore), the server returns an HTTP 400 (Bad Request) error: + +```text +GET /puppet/v3/environment_modules?environment=bog|us + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not 'bog|us' +``` + +### No metadata.json file + +If your modules do not have a [metadata.json](/openvox/latest/modules_metadata.html) +file, OpenVox Server will not be able to determine the version of your module. +In this case, OpenVox Server will return a null value for `version` in the response body. + +### Schema + +An environment modules response body conforms to the +[environment modules schema](./environment_modules.json). + +#### Validating your json + +If you have a response body that you'd like to validate against the +[environment_modules.json](./environment_modules.json) schema, you can do so using the ruby library +[json-schema](https://github.com/ruby-json-schema/json-schema). + +First, install the ruby gem to be used: + +```bash +gem install json-schema +``` + +Next, given a json file, you can validate its schema. + +Here is a basic json file called _example.json_: + +```json +{ + "modules": [ + { + "name": "puppet/systemd", + "version": "9.4.0" + }, + { + "name": "puppetlabs/stdlib", + "version": "9.7.0" + } + ], + "name": "production" +} +``` + +Run this command from the root dir of the OpenVox Server project (or update the path to the json schema file in the command below): + +```bash +ruby -rjson-schema -e "puts JSON::Validator.validate!('./documentation/puppet-api/v3/environment_modules.json','example.json')" +``` + +If the json is a valid schema, the command should output `true`. Otherwise, the library will print a schema validation error detailing which key or keys validate the schema. + +If you have a response that is the entire list of environment modules (i.e. the environment_modules endpoint), you will need to use this command to validate the json schema: + +```bash +ruby -rjson-schema -e "puts JSON::Validator.validate!('./documentation/puppet-api/v3/environment_modules.json','all.json', :list=>true)" +``` + +### Authorization + +All requests made to the environment classes API are authorized using the Trapperkeeper-based [`auth.conf`][]. + +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][`auth.conf`]. diff --git a/docs/_openvox-server_9x/puppet-api/v3/environment_transports.json b/docs/_openvox-server_9x/puppet-api/v3/environment_transports.json new file mode 100644 index 000000000..81f77627f --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/environment_transports.json @@ -0,0 +1,52 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Environment Transports", + "description": "Information about the Resource API Transports in a Puppet code environment", + "type": "object", + "additionalProperties": false, + "properties": { + "name": { + "description": "The name of the environment queried", + "type": "string" + }, + "transports": { + "description": "A list of available transports for this environment", + "type": "array", + "items": { + "type": "object", + "additonalProperties": false, + "properties": { + "name": { + "description": "The machine readable name of the transport", + "type": "string" + }, + "desc": { + "description": "The human readable description of the transport this schema specifies", + "type": "string" + }, + "connection_info": { + "type": "object", + "additionalProperties": { + "type": "object", + "required": ["type", "desc"], + "properties": { + "type": { + "description": "The PCore type of the connection info property", + "type": "string" + }, + "desc": { + "description": "The description of the connection info property", + "type": "string" + }, + "sensitive": { + "description": "Whether or not the connection info property should be considered sensitive", + "type": "boolean" + } + } + } + } + } + } + } + } +} diff --git a/docs/_openvox-server_9x/puppet-api/v3/environment_transports.markdown b/docs/_openvox-server_9x/puppet-api/v3/environment_transports.markdown new file mode 100644 index 000000000..6146b77de --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/environment_transports.markdown @@ -0,0 +1,30 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Environment Transports" +--- + +[HTTP API]: /openvox-server/latest/http_api_index.html +[environment cache API]: ../../admin-api/v1/environment-cache.html +[environment classes API]: ./environment_classes.html +[transports schema]: ./environment_transports.json +[`auth.conf` documentation]: ../../config_file_auth.html + +The environment transports API returns a JSON object representing the requested environment and schemas for all available [HTTP API][] endpoints. +The endpoint follows all conventions set by the [environment classes API][] including request format, etag validation with expiration managed by the [environment cache API][], and errors. + +## `GET /puppet/v3/environment_transports?environment=` + +### Query Parameters + +#### `environment` (required) + +The name of the environment to query for available device transport schemas. + +### Schema + +The transports endpoint response body conforms to the [transports schema][]. + +### Authorization + +All requests made to the environment transports API are authorized using the Trapperkeeper-based `auth.conf`. +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][]. diff --git a/docs/_openvox-server_9x/puppet-api/v3/file_content.markdown b/docs/_openvox-server_9x/puppet-api/v3/file_content.markdown new file mode 100644 index 000000000..e736b64e6 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/file_content.markdown @@ -0,0 +1,28 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: File Content" +--- + +The `file_content` endpoint returns contents of the specified file. + +## `GET /puppet/v3/file_content/:mount_point/:module/:file-path?environment=:environment` + +When specifying environment see the [OpenVox API docs](../../http_file_content.html). + +## `GET /puppet/v3/file_content/:mount_point/:module/:file-path?project=:project-ref` + +Return the contents of a file from the project specified by :project-ref, which should be of the form :name-:version + +The `:mount_point` specifies where to look inside modules to find the `:file-path`. + +- `modules` - Find `:file-path` under the `files` subdirectory of `:module` +- `tasks` - Find `:file-path` under the `tasks` subdirectory of `:module` + +### Response + +A successful request to this endpoint returns an `HTTP 200` response code and `application/octet-stream` Content-Type header, and the contents of the specified file in the response body. +An unsuccessful request returns an error response +code with a `text/plain` Content-Type header: + +- 400: returned when neither the environment nor the project query parameters are provided, or when both project and environment are provided. +- 404: returned when requesting a file that is not within a module's `files` or `tasks` directory, or when any other component (project, module, or mount point) is not found. diff --git a/docs/_openvox-server_9x/puppet-api/v3/plan_detail.markdown b/docs/_openvox-server_9x/puppet-api/v3/plan_detail.markdown new file mode 100644 index 000000000..755da6619 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/plan_detail.markdown @@ -0,0 +1,128 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Plan detail" +--- + +## Uses `application/json` Content-Type + +The Content-Type in the response to an plan API query is +`application/json`. + +## `GET /puppet/v3/plans/:module/:plan?environment=:environment` + +Making a request with no query parameters is not supported and returns an HTTP 400 (Bad Request) response. + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Query Parameters + +Provide one parameter to the GET request: + +* `environment`: Only the plan information pertaining to the specified environment will be returned for the call. + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/plans/module/planname?environment=env + +HTTP/1.1 200 OK +Content-Type: application/json;charset=utf-8 + +{ + "metadata": {}, + "name": "module::planname" +} +``` + +#### GET request for invalid module + +If you request details for a plan which cannot be computed because the metadata is unreadable or it's implementations are not usable Bolt will return an error response +with a status code of 500 containing `kind`, `msg`, and `details` keys. + +```text +GET /puppet/v3/plans/modulename/planname?environment=env +HTTP/1.1 500 Server Error +Content-Type: application/json + +{ + "details": {}, + "kind": "puppet.plans/unparseable-metadata", + "msg": "unexpected token at '{ \"name\": \"init.sh\" , }\n ]\n}\n'" +} +``` + +#### Environment does not exist + +If you send a request with an environment parameter that doesn't correspond to the name of a directory environment on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/plans/module/planname?environment=doesnotexist + +HTTP/1.1 404 Not Found + +Could not find environment 'doesnotexist' +``` + +#### No environment given + +```text +GET /puppet/v3/plans/module/planname + +HTTP/1.1 400 Bad Request + +An environment parameter must be specified +``` + +#### Environment parameter specified with no value + +```text +GET /puppet/v3/plans/module/planname?environment= + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not '' +``` + +#### Environment includes non-alphanumeric characters + +If the environment parameter in your request includes any characters that are not `A-Z`, `a-z`, `0-9`, or `_` (underscore), the server returns an HTTP 400 (Bad Request) error: + +```text +GET /puppet/v3/plans/module/planname?environment=bog|us + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not 'bog|us' +``` + +#### Module does not exist + +If you send a request for a plan in a module that doesn't correspond to the name of a module on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/plans/doesnotexist/planname?environment=env + +HTTP/1.1 404 Not Found + +Could not find module 'doesnotexist' +``` + +#### Plan does not exist or does not have a valid name + +If you send a request for a plan in that doesn't correspond to the name of a plan on the server, but the module does exist, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/plans/module/doesnotexist?environment=env + +HTTP/1.1 404 Not Found + +Could not find plan 'doesnotexist' +``` diff --git a/docs/_openvox-server_9x/puppet-api/v3/plans.markdown b/docs/_openvox-server_9x/puppet-api/v3/plans.markdown new file mode 100644 index 000000000..710a810d7 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/plans.markdown @@ -0,0 +1,112 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Plans" +--- + +## Uses `application/json` Content-Type + +The Content-Type in the response to an plan API query is +`application/json`. + +## `GET /puppet/v3/plans?environment=:environment` + +Making a request with no query parameters is not supported and returns an HTTP 400 (Bad Request) response. + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Query Parameters + +Provide one parameter to the GET request: + +* `environment`: Only the plan information pertaining to the specified environment will be returned for the call. + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/plans?environment=env + +HTTP/1.1 200 OK +Content-Type: application/json;charset=utf-8 + +[ + { + "name": "apache::init", + "environment": [ + { + "name": "production", + "code_id": null + } + ] + }, + { + "name": "apache::announce", + "environment": [ + { + "name": "production", + "code_id": null + } + ] + }, + { + "name": "graphite", + "environment": [ + { + "name": "production", + "code_id": null + } + ] + } +] +``` + +#### Environment does not exist + +If you send a request with an environment parameter that doesn't correspond to the name of a directory environment on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/plans?environment=doesnotexist + +HTTP/1.1 404 Not Found + +Could not find environment 'doesnotexist' +``` + +#### No environment given + +```text +GET /puppet/v3/plans + +HTTP/1.1 400 Bad Request + +An environment parameter must be specified +``` + +#### Environment parameter specified with no value + +```text +GET /puppet/v3/plans?environment= + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not '' +``` + +#### Environment includes non-alphanumeric characters + +If the environment parameter in your request includes any characters that are not `A-Z`, `a-z`, `0-9`, or `_` (underscore), the server returns an HTTP 400 (Bad Request) error: + +```text +GET /puppet/v3/plans?environment=bog|us + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not 'bog|us' +``` diff --git a/docs/_openvox-server_9x/puppet-api/v3/static_file_content.markdown b/docs/_openvox-server_9x/puppet-api/v3/static_file_content.markdown new file mode 100644 index 000000000..a71a10f6a --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/static_file_content.markdown @@ -0,0 +1,82 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Static File Content" +--- + +[`code-content-command`]: ../../config_file_puppetserver.html +[static catalog]: /openvox/latest/static_catalogs.html +[catalog]: /openvox/latest/subsystem_catalog_compilation.html +[file resource]: /openvox/latest/type.html#file +[environment]: /openvox/latest/environments_about.html +[`auth.conf`]: ../../config_file_auth.html + +The `static_file_content` endpoint returns the standard output of a +[`code-content-command`][] script, which should output the contents of a specific version of a [file resource][] that has a `source` attribute with a `puppet:///` URI value. +That source must be a file from the `files` or `tasks` directory of a module in a specific [environment][]. + +OpenVox Agent uses this endpoint only when applying a [static catalog][]. + +## `GET /puppet/v3/static_file_content/` + +To retrieve a specific version of a file at a given environment and path, make an HTTP request to this endpoint with the required parameters. + +The `` segment of the endpoint is required. The path corresponds to the requested file's path on the Server relative to the given environment's root directory, +and must point to a file in the `*/*/files/**`, `*/*/lib/**`, or `*/*/tasks/**` glob. +For example, OpenVox Server will inline metadata into static catalogs for file resources sourcing module files located by default in `/etc/puppetlabs/code/environments//modules//files/**`. + +### Query parameters + +You must also pass two parameters in the GET request: + +- `code_id`: a unique string provided by the [catalog][] that identifies which version of the file to return. +- `environment`: the environment that contains the desired file. + +### Response + +A successful request to this endpoint returns an `HTTP 200` response code and `application/octet-stream` Content-Type header, and the contents of the specified file's requested version in the response body. +An unsuccessful request returns an error response code with a `text/plain` Content-Type header: + +- 400: returned when any of the parameters are not provided. +- 403: returned when requesting a file that is not within a module's `files` or `tasks` directory. +- 500: returned when `code-content-command` is not configured on the server, or when a requested file or version is not present in a repository. + +#### Example response + +Consider a server `localhost`, with a versioned file located at +`/modules/example/files/data.txt` in the `production` environment. +The version is identified by a `code_id` of `urn:puppet:code-id:1:67eb71417fbd736a619c8b5f9bfc0056ea8c53ca;production`, and that version of the file contains `Puppet test`. + +If you run this command: + +```command +curl -i -k 'https://localhost:8140/puppet/v3/static_file_content/modules/example/files/data.txt?code_id=urn:puppet:code-id:1:67eb71417fbd736a619c8b5f9bfc0056ea8c53ca;production&environment=production' +``` + +OpenVox Server returns: + +```text +HTTP/1.1 200 OK +Date: Wed, 2 Mar 2016 23:44:08 GMT +X-Puppet-Version: 4.4.0 +Content-Length: 4 +Server: Jetty(9.2.10.v20150310) + +Puppet test +``` + +### Notes + +When requesting a file from this endpoint, OpenVox Server passes the values of the `file-path`, `code_id`, and `environment` parameters as arguments to the `code-content-command` script. +If the script returns an exit code of 0, OpenVox Server returns the script's standard output, which should be the contents of the requested version of the file. + +This endpoint returns an error (status 500) if the [`code-content-command`][] setting is not configured on OpenVox Server. + +> **Note:** The `code-content-command` and `code-id-command` scripts are not provided in a +> default installation or upgrade. For more information about these scripts, see the +> [static catalog documentation](/openvox/latest/static_catalogs.html). + +#### Authorization + +All requests made to the environment classes API are authorized using the Trapperkeeper-based [`auth.conf`][]. + +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][`auth.conf`]. diff --git a/docs/_openvox-server_9x/puppet-api/v3/task_detail.json b/docs/_openvox-server_9x/puppet-api/v3/task_detail.json new file mode 100644 index 000000000..88ff5f6a4 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/task_detail.json @@ -0,0 +1,63 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Task detail", + "description": "Detailed information about a specific task", + "type": "object", + "properties": { + "metadata": { + "description": "The contents of the .json metadata file. This will be empty if the file doesn't exist, or is empty. Otherwise it is allowed to return arbitrary keys here.", + "type": "object", + "properties": {}, + "required": [], + "additionalProperties": true + }, + "files": { + "description": "A list of executable files associated with the requested task, and data about them. The data is intended to assist in fetching their contents from the task file-content endpoint.", + "type": "array", + "items": { + "description": "A single file associated with the requested task, and data about it.", + "type": "object", + "properties": { + "filename": { + "description": "The name of the file (which should match the second part of the task name) plus its extension.", + "type": "string" + }, + "sha256": { + "description": "The SHA256 of the contents of the file.", + "type": "string" + }, + "size_bytes": { + "description": "The size of the file, in bytes.", + "type": "number" + }, + "uri": { + "description": "Information on how to request the file contents from an OpenVox Server node. This will only provide a relative path, because clients may want to request from a compile server instead of the primary server.", + "type": "object", + "properties": { + "path": { + "description": "A relative URI for accessing the task contents. A client can form the full URI by requesting 'server:port/path?params'.", + "type": "string" + }, + "params": { + "description": "A map of query params to use when requesting task contents. Required to include in the URI when making a request for task contents.", + "type": "object", + "properties": { + "environment": { + "description": "The environment the task is in, a required query param on request to task contents.", + "type": "string" + } + }, + "required": ["environment"], + "additionalProperties": true + } + }, + "required": ["path", "params"], + "additionalProperties": true + } + }, + "required": ["metadata", "files", "code_id"], + "additionalProperties": true + } + } + } +} diff --git a/docs/_openvox-server_9x/puppet-api/v3/task_detail.markdown b/docs/_openvox-server_9x/puppet-api/v3/task_detail.markdown new file mode 100644 index 000000000..e063ab8db --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/task_detail.markdown @@ -0,0 +1,181 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Task detail" +--- + +[`auth.conf`]: ../../config_file_auth.html + +The tasks API provides access to task information stored in modules. Tasks are files stored in `tasks` subdirectory of a module. +A task consists of an executable file, with an optional metadata file with the same name with an added '.json' extension. +For example, the "install" task in a module "apache" could consist of the executable file `install.rb` and the metadata file `install.json`. +This task would have the display name "apache::install". + +This endpoint, `/puppet/v3/tasks/:module/:taskname`, allows you to fetch the details about a task: its metadata, if present, and its associated executable files. +The file entries have additional data on how to fetch their contents so they can be downloaded and run. + +> Note: Tasks file contents in versioned code can be retrieved using the [`static_file_content`](./static_file_content.html) endpoint. + +## Does not return entries for task files with invalid names + +A task file name has the same restriction as puppet type names and must match the regular expression `\A[a-z][a-z0-9_]*\z` (excluding extensions). + +## Will error if the tasks implementations are invalid + +Because the returning file information requires parsing metadata and finding implementation files this endpoint will error if the metadata cannot be parsed or the implementation content is invalid. + +## Does read files + +This endpoint will read in contents of metadata and other task files, so it may be more expensive than the `/tasks` endpoint. + +## Uses `application/json` Content-Type + +The Content-Type in the response to an task API query is `application/json`. + +## `GET /puppet/v3/tasks/:module/:task?environment=:environment` + +Making a request with no query parameters is not supported and returns an HTTP 400 (Bad Request) response. + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Query Parameters + +Provide one parameter to the GET request: + +- `environment`: Only the task information pertaining to the specified environment will be returned for the call. + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/tasks/module/taskname?environment=env + +HTTP/1.1 200 OK +Etag: b02ede6ecc432b134217a1cc681c406288ef9224 +Content-Type: application/json + +{ + "metadata": { + "description": "Install a package", + "parameters": { + "name": { + "description": "The package to install", + "type": "String[1]" + } + } + }, + "files": [ + {"filename": "taskname.rb", + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "size_bytes": 1024, + "uri:" { + "path": "/puppet/v3/file_content/tasks/module/taskname.rb", + "params": { + "environment": "production" + } + } + } + ] +} +``` + +#### GET request for invalid module + +If you request details for a task which cannot be computed because the metadata is unreadable or it's implementations are not usable Bolt will return an error response with a status code of 500 containing +`kind`, `msg`, and `details` keys. + +```text +GET /puppet/v3/tasks/modulename/taskname?environment=env +HTTP/1.1 500 Server Error +Content-Type: application/json + +{ + "details": {}, + "kind": "puppet.tasks/unparseable-metadata", + "msg": "unexpected token at '{ \"name\": \"init.sh\" , }\n ]\n}\n'" +} +``` + +#### Environment does not exist + +If you send a request with an environment parameter that doesn't correspond to the name of a directory environment on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/tasks/module/taskname?environment=doesnotexist + +HTTP/1.1 404 Not Found + +Could not find environment 'doesnotexist' +``` + +#### No environment given + +```text +GET /puppet/v3/tasks/module/taskname + +HTTP/1.1 400 Bad Request + +You must specify an environment parameter. +``` + +#### Environment parameter specified with no value + +```text +GET /puppet/v3/tasks/module/taskname?environment= + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not '' +``` + +#### Environment includes non-alphanumeric characters + +If the environment parameter in your request includes any characters that are not `A-Z`, `a-z`, `0-9`, or `_` (underscore), the server returns an HTTP 400 (Bad Request) error: + +```text +GET /puppet/v3/tasks/module/taskname?environment=bog|us + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not 'bog|us' +``` + +#### Module does not exist + +If you send a request for a task in a module that doesn't correspond to the name of a module on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/tasks/doesnotexist/taskname?environment=env + +HTTP/1.1 404 Not Found + +Could not find module 'doesnotexist' +``` + +#### Task does not exist or does not have a valid name + +If you send a request for a task in that doesn't correspond to the name of a task on the server, but the module does exist, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/tasks/module/doesnotexist?environment=env + +HTTP/1.1 404 Not Found + +Could not find task 'doesnotexist' +``` + +### Schema + +A tasks detail response body conforms to the [task detail schema](./task_detail.json). + +### Authorization + +All requests made to the environment classes API are authorized using the Trapperkeeper-based [`auth.conf`][]. + +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][`auth.conf`]. diff --git a/docs/_openvox-server_9x/puppet-api/v3/tasks.json b/docs/_openvox-server_9x/puppet-api/v3/tasks.json new file mode 100644 index 000000000..9f567eb4d --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/tasks.json @@ -0,0 +1,38 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Tasks list", + "description": "List of task objects in a Puppet code environment", + "type": "array", + "items": { + "title": "Task", + "type": "object", + "properties": { + "name": { + "description": "Name of the task object. Takes the form of '::'. Tasks with invalid characters in their files' names will not be listed.", + "type": "string" + }, + "environment": { + "description": "A list of environments the task was found in. Will only contain one object when a specific environment is requested, which is all that is currently supported.", + "type": "array", + "items": { + "description": "The environment that the task was found in.", + "type": "object", + "properties": { + "name": { + "description": "The name of the environment the task was found in.", + "type": "string" + }, + "code_id": { + "description": "The code_id for the environment, if it exists. Not yet implemented, so will always return null.", + "type": "null" + } + }, + "required": ["name"], + "additionalProperties": true + } + } + }, + "required": ["name", "environment"], + "additionalProperties": true + } +} diff --git a/docs/_openvox-server_9x/puppet-api/v3/tasks.markdown b/docs/_openvox-server_9x/puppet-api/v3/tasks.markdown new file mode 100644 index 000000000..6e1b7b044 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v3/tasks.markdown @@ -0,0 +1,133 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Tasks" +--- + +[`auth.conf`]: ../../config_file_auth.html + +The tasks API provides access to task information stored in modules. Tasks are files stored in `tasks` subdirectory of a module. +A task consists of an executable file, with an optional metadata file with the same name with an added '.json' extension. +For example, the "install" task in a module "apache" could consist of the executable file `install.rb` and the metadata file `install.json`. This task would have the display name "apache::install". + +> Note: Tasks file contents in versioned code can be retrieved using the [`static_file_content`](./static_file_content.html) endpoint. + +## Does not return entries for task files with invalid names + +A task file name has the same restriction as Puppet type names and must match the regular expression `\A[a-z][a-z0-9_]*\z` (excluding extensions). + +## Returns entries for tasks with no executable files + +A task will be listed if only metadata for it exists. How many files are associated with a task can be found by querying that task's details. + +## Does not read files + +This endpoint will not parse metadata or read any other files, only file names. + +## Uses `application/json` Content-Type + +The Content-Type in the response to an task API query is `application/json`. + +## `GET /puppet/v3/tasks?environment=:environment` + +Making a request with no query parameters is not supported and returns an HTTP 400 (Bad Request) response. + +### Supported HTTP Methods + +GET + +### Supported Formats + +JSON + +### Query Parameters + +Provide one parameter to the GET request: + +- `environment`: Only the task information pertaining to the specified environment will be returned for the call. + +### Responses + +#### GET request with results + +```text +GET /puppet/v3/tasks?environment=env + +HTTP/1.1 200 OK +Etag: b02ede6ecc432b134217a1cc681c406288ef9224 +Content-Type: application/json + +[ + { + "name":"apache::install", + "environment":[ + { + "name":"env", + "code_id":null + } + ] + }, + { + "name":"dashboard::configure", + "environment":[ + { + "name":"env", + "code_id":null + } + ] + } +] +``` + +#### Environment does not exist + +If you send a request with an environment parameter that doesn't correspond to the name of a directory environment on the server, the server returns an HTTP 404 (Not Found) error: + +```text +GET /puppet/v3/tasks?environment=doesnotexist + +HTTP/1.1 404 Not Found + +Could not find environment 'doesnotexist' +``` + +#### No environment given + +```text +GET /puppet/v3/tasks + +HTTP/1.1 400 Bad Request + +You must specify an environment parameter. +``` + +#### Environment parameter specified with no value + +```text +GET /puppet/v3/tasks?environment= + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not '' +``` + +#### Environment includes non-alphanumeric characters + +If the environment parameter in your request includes any characters that are not `A-Z`, `a-z`, `0-9`, or `_` (underscore), the server returns an HTTP 400 (Bad Request) error: + +```text +GET /puppet/v3/tasks?environment=bog|us + +HTTP/1.1 400 Bad Request + +The environment must be purely alphanumeric, not 'bog|us' +``` + +### Schema + +A tasks response body conforms to the [tasks schema](./tasks.json). + +### Authorization + +All requests made to the environment classes API are authorized using the Trapperkeeper-based [`auth.conf`][]. + +For more information about the OpenVox Server authorization process and configuration settings, see the [`auth.conf` documentation][`auth.conf`]. diff --git a/docs/_openvox-server_9x/puppet-api/v4/catalog.json b/docs/_openvox-server_9x/puppet-api/v4/catalog.json new file mode 100644 index 000000000..c6fae4b73 --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v4/catalog.json @@ -0,0 +1,81 @@ +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "title": "Catalog", + "description": "Serialization of the Puppet catalog for a given node", + "type": "object", + "properties": { + "catalog": { + "type": "object", + "$ref": "https://docs.openvoxproject.org/openvox/latest/schemas/catalog.json" + }, + "logs": { + "description": "The log messages that occured during the catalog compilation.", + "type": "array", + "items": { + "type": "object", + "$ref": "#/definitions/log" + } + } + }, + "required": ["catalog"], + "additionalProperties": false, + + "definitions" : { + "log" : { + "properties": { + "file": { + "description": "The pathname of the manifest file which triggered the log message.", + "oneOf": [ + {"type": "string"}, + {"type": "null"} + ] + }, + "line": { + "description": "The line number in the manifest file which triggered the log message.", + "oneOf": [ + {"type": "string"}, + {"type": "null"} + ] + }, + "level": { + "description": "The severity of the message.", + "enum": [ + "debug", + "info", + "notice", + "warning", + "err", + "alert", + "emerg", + "crit" + ] + }, + "message": { + "description": "The message itself.", + "type": "string" + }, + "source": { + "description": "The origin of the log message. This could be a resource, a property of a resource, or the string 'Puppet'.", + "type": "string" + }, + "tags": { + "description": "The strings with which the source is tagged", + "type": "array", + "items": { "type": "string" } + }, + "time": { + "description": "When the message was sent. In ISO 8601 format with 9 characters second-fragment", + "type": "string" + } + } + }, + "required": [ + "level", + "message", + "source", + "tags", + "time" + ], + "additionalProperties": false + } +} diff --git a/docs/_openvox-server_9x/puppet-api/v4/catalog.markdown b/docs/_openvox-server_9x/puppet-api/v4/catalog.markdown new file mode 100644 index 000000000..8e53432bd --- /dev/null +++ b/docs/_openvox-server_9x/puppet-api/v4/catalog.markdown @@ -0,0 +1,77 @@ +--- +layout: default +title: "OpenVox Server: Puppet API: Catalog" +--- + +The catalog API returns a compiled catalog for the node specified in the request, making use of provided metadata like facts or environment if specified. If not specified, it will attempt to fetch this data +from OpenVox's configured sources (usually OpenVox-DB or a node classifier). The returned catalog is in JSON format, ready to be parsed and applied by an agent. + +## `POST /puppet/v4/catalog` + +The input data for the catalog to be compiled is submitted as a JSON body with the following form: + +```text +{ + "certname": "", + "persistence": { "facts": , "catalog": }, + "environment": "", + # The rest are optional: + "facts": { "values": { "": , ... } }, + "trusted_facts": { "values": { "": , ... } }, + "transaction_uuid": "", + "job_id": "", + "options": { "prefer_requested_environment": , + "capture_logs": , + "log_level": } +} +``` + +### `certname` (required) + +The name of the node for which to compile the catalog. + +### `persistence` (required) + +A hash containing two required keys, `facts` and `catalog`, which when set to true will cause the facts and reports to be stored in OpenVox-DB, or discarded if set to false. + +### `environment` (required) + +The name of the environment for which to compile the catalog. If `prefer_requested_environemnt` is true, override the classified environment with this param. If it is false, only respect this if the classifier +allows an agent-specified environment. + +### `facts` + +A hash with a required `values` key, containing a hash of all the facts for the node. If not provided, Puppet will attempt to fetch facts for the node from OpenVox-DB. + +### `trusted_facts` + +A hash with a required `values` key containing a hash of the trusted facts for a node. In a normal agent's catalog request, these would be extracted from the cert, but this endpoint does not require a cert for +the node whose catalog is being compiled. If not provided, OpenVox will attempt to fetch the trusted facts for the node from OpenVox-DB or from the provided facts hash. + +### `transaction_uuid` + +The id for tracking the catalog compilation and report submission. + +### `job_id` + +The id of the orchestrator job that triggered this run. + +### `options` + +A hash of options beyond direct input to catalogs. + +`prefer_requested_environment` Whether to always override a node's classified environment with the one supplied in the request. If this is true and no environment is supplied, fall back to the classified +environment, or finally, 'production'. + +`capture_logs` Whether to return the errors and warnings that occurred during compilation alongside the catalog in the response body. + +`log_level` The logging level to use during the compile when `capture_logs` is true. Options are 'err', 'warning', 'info', and 'debug'. + +### Schema + +The catalog response body conforms to the [catalog schema](./catalog.json). + +### Authorization + +All requests made to the catalog API are authorized using the Trapperkeeper-based `auth.conf`. For more information about the OpenVox Server authorization process and configuration settings, see the +[`auth.conf` documentation](../../config_file_auth.html). diff --git a/docs/_openvox-server_9x/puppet_conf_setting_diffs.markdown b/docs/_openvox-server_9x/puppet_conf_setting_diffs.markdown new file mode 100644 index 000000000..96a970d7b --- /dev/null +++ b/docs/_openvox-server_9x/puppet_conf_setting_diffs.markdown @@ -0,0 +1,115 @@ +--- +layout: default +title: "How OpenVox Server uses the values in puppet.conf" +--- + +OpenVox Server honors almost all settings in puppet.conf and should pick them up automatically. For more complete information on puppet.conf settings, see the +[Configuration Reference](/openvox/latest/configuration.html) page. + +## `ca_ttl` + +OpenVox Server enforces a max ttl of 50 standard years (up to 1576800000 seconds). + +## `cacert` + +If you enable OpenVox Server's certificate authority service, it uses the `cacert` setting in puppet.conf to determine the location of the CA certificate for such tasks as generating the CA certificate or +using the CA to sign client certificates. This is true regardless of the configuration of the `ssl-` settings in [webserver.conf](./config_file_webserver.html). + +## `cacrl` + +If you define `ssl-cert`, `ssl-key`, `ssl-ca-cert`, or `ssl-crl-path` in [webserver.conf](./config_file_webserver.html), OpenVox Server uses the file at `ssl-crl-path` as the CRL for authenticating +clients via SSL. If at least one of the `ssl-` settings in webserver.conf is set but `ssl-crl-path` is not set, OpenVox Server will _not_ use a CRL to validate clients via SSL. + +If none of the `ssl-` settings in webserver.conf are set, OpenVox Server uses the CRL file defined for the `hostcrl` setting---and not the file defined for the `cacrl` setting--in puppet.conf. At start time, +OpenVox Server copies the file for the `cacrl` setting, if one exists, over to the location in the `hostcrl` setting. + +Any CRL file updates from the OpenVox Server certificate authority---such as revocations performed via the `certificate_status` HTTP endpoint---use the `cacrl` setting in puppet.conf to determine the location +of the CRL. This is true regardless of the `ssl-` settings in webserver.conf. + +## `hostcert` + +If you define `ssl-cert`, `ssl-key`, `ssl-ca-cert`, or `ssl-crl-path` in [webserver.conf](./config_file_webserver.html), OpenVox Server presents the file at `ssl-cert` to clients as the server +certificate via SSL. + +If at least one of the `ssl-` settings in webserver.conf is set but `ssl-cert` is not set, OpenVox Server gives an error and shuts down at startup. If none of the `ssl-` settings in webserver.conf are set, +OpenVox Server uses the file for the `hostcert` setting in puppet.conf as the server certificate during SSL negotiation. + +Regardless of the configuration of the `ssl-` `webserver.conf` settings, OpenVox Server's certificate authority service, if enabled, uses the `hostcert` `puppet.conf` setting, and not the `ssl-cert` setting, +to determine the location of the server host certificate to generate. + +## `hostcrl` + +If you define `ssl-cert`, `ssl-key`, `ssl-ca-cert`, or `ssl-crl-path` in [webserver.conf](./config_file_webserver.html), OpenVox Server uses the file at `ssl-crl-path` as the CRL for authenticating +clients via SSL. If at least one of the `ssl-` settings in webserver.conf is set but `ssl-crl-path` is not set, OpenVox Server will _not_ use a CRL to validate clients via SSL. + +If none of the `ssl-` settings in webserver.conf are set, OpenVox Server uses the CRL file defined for the `hostcrl` setting---and not the file defined for the `cacrl` setting--in puppet.conf. At start time, +OpenVox Server copies the file for the `cacrl` setting, if one exists, over to the location in the `hostcrl` setting. + +Any CRL file updates from the OpenVox Server certificate authority---such as revocations performed via the `certificate_status` HTTP endpoint---use the `cacrl` setting in puppet.conf to determine the location +of the CRL. This is true regardless of the `ssl-` settings in webserver.conf. + +## `hostprivkey` + +If you define `ssl-cert`, `ssl-key`, `ssl-ca-cert`, or `ssl-crl-path` in [webserver.conf](./config_file_webserver.html), OpenVox Server uses the file at `ssl-key` as the server private key during SSL +transactions. + +If at least one of the `ssl-` settings in webserver.conf is set but `ssl-key` is not, OpenVox Server gives an error and shuts down at startup. If none of the `ssl-` settings in webserver.conf are set, +OpenVox Server uses the file for the `hostprivkey` setting in puppet.conf as the server private key during SSL negotiation. + +If you enable the OpenVox Server certificate authority service, OpenVox Server uses the `hostprivkey` setting in puppet.conf to determine the location of the server host private key to generate. This is true +regardless of the configuration of the `ssl-` settings in webserver.conf. + +## `localcacert` + +If you define `ssl-cert`, `ssl-key`, `ssl-ca-cert`, and/or `ssl-crl-path` in [webserver.conf](./config_file_webserver.html), OpenVox Server uses the file at `ssl-ca-cert` as the CA cert store for +authenticating clients via SSL. + +If at least one of the `ssl-` settings in webserver.conf is set but `ssl-ca-cert` is not set, OpenVox Server gives an error and shuts down at startup. If none of the `ssl-` settings in webserver.conf is set, +OpenVox Server uses the CA file defined for the `localcacert` setting in puppet.conf for SSL authentication. + +## `masterport` + +OpenVox Server does not use this setting. To set the port on which the server listens, set the `port` (unencrypted) or `ssl-port` (SSL encrypted) setting in the +[webserver.conf](./config_file_webserver.html) file. + +## `ssl_client_header` + +OpenVox Server honors this setting only if the `allow-header-cert-info` setting in the [`master.conf`](./config_file_master.html) file (deprecated) is set to `true`. For more information, +see the documentation on [external SSL termination](./external_ssl_termination.html). + +## `ssl_client_verify_header` + +OpenVox Server honors this setting only if the `allow-header-cert-info` setting in the [`master.conf`](./config_file_master.html) file (deprecated) is set to `true`. For more information, +see the documentation on [external SSL termination](./external_ssl_termination.html). + +## HttpPool-Related Server Settings + +## `http_proxy_host` + +OpenVox Server does not currently consider this setting for any code running on the server and using the `Puppet::Network::HttpPool` module to create an HTTP client connection. This pertains, for example, to +any requests that the server would make to the `reporturl` for the `http` report processor. Note that Puppet agents do still honor this setting. + +## `http_proxy_port` + +OpenVox Server does not currently consider this setting for any code running on the server and using the `Puppet::Network::HttpPool` module to create an HTTP client connection. This pertains, for example, to +any requests that the server would make to the `reporturl` for the `http` report processor. Note that Puppet agents do still honor this setting. + +## Overriding Puppet settings in OpenVox Server + +Currently, the [`jruby-puppet` section of your `puppetserver.conf` file](./config_file_puppetserver.html) contains five settings (`master-conf-dir`, `master-code-dir`, `master-var-dir`, +`master-run-dir`, and `master-log-dir`) that allow you to override settings set in your `puppet.conf` file. On installation, these five settings will be set to the proper default values. + +While you are free to change these settings at will, please note that any changes made to the `master-conf-dir` and `master-code-dir` settings absolutely MUST be made to the corresponding Puppet settings +(`confdir` and `codedir`) as well to ensure that OpenVox Server and the Puppet cli tools (such as `puppetserver ca` and `puppet module`) use the same directories. The `master-conf-dir` and `master-code-dir` +settings apply to OpenVox Server only, and will be ignored by the ruby code that runs when the Puppet CLI tools are run. + +For example, say you have the `codedir` setting left unset in your `puppet.conf` file, and you change the `master-code-dir` setting to `/etc/my-puppet-code-dir`. In this case, OpenVox Server will read code +from `/etc/my-puppet-code-dir`, but the `puppet module` tool will think that your code is stored in `/etc/puppetlabs/code`. + +While it is not as critical to keep `master-var-dir`, `master-run-dir`, and `master-log-dir` in sync with the `vardir`, `rundir`, and `logdir` Puppet settings, please note that this applies to these settings +as well. + +Also, please note that these configuration differences also apply to the interpolation of the `confdir`, `codedir`, `vardir`, `rundir`, and `logdir` settings in your `puppet.conf` file. So, take the above +example, wherein you set `master-code-dir` to `/etc/my-puppet-code-dir`. Because the `basemodulepath` setting is by default `$codedir/modules:/opt/puppetlabs/puppet/modules`, then OpenVox Server would use +`/etc/my-puppet-code-dir/modules:/opt/puppetlabs/puppet/modules` for the value of the `basemodulepath` setting, whereas the `puppet module` tool would use +`/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules` for the value of the `basemodulepath` setting. diff --git a/docs/_openvox-server_9x/puppet_server_metrics.markdown b/docs/_openvox-server_9x/puppet_server_metrics.markdown new file mode 100644 index 000000000..46f1ef706 --- /dev/null +++ b/docs/_openvox-server_9x/puppet_server_metrics.markdown @@ -0,0 +1,452 @@ +--- +layout: default +title: "Monitoring OpenVox Server metrics" +--- + +[metrics API]: ./metrics-api/v1/metrics_api.html +[Graphite]: https://graphiteapp.org +[Grafana]: http://grafana.org +[sample Grafana dashboard]: ./sample-puppetserver-metrics-dashboard.json +[static catalogs]: /openvox/latest/static_catalogs.html +[HTTP client metrics]: ./http_client_metrics.html +[`grafanadash`]: https://forge.puppet.com/cprice404/grafanadash +[`metrics.conf`]: ./config_file_metrics.html + +OpenVox Server tracks several advanced performance and health metrics, all of which take advantage of the [metrics API][]. You can track these metrics using: + +- Customizable, networked [Graphite and Grafana instances](#getting-started-with-graphite) +- [HTTP client metrics][] +- [Metrics API][metrics API] endpoints + +To visualize OpenVox Server metrics, either: + +- Use the [puppet-operational-dashboards](https://forge.puppet.com/puppetlabs/puppet_operational_dashboards) module. +- Export them to a Graphite installation. The [grafanadash](https://forge.puppet.com/puppetlabs/grafanadash) module helps you set up a Graphite instance, configure OpenVox Server for exporting to it, and + visualize the output with Grafana. You can later integrate this with your Graphite installation. For more information, see Getting started with Graphite below. + +The puppet-operational-dashboards module is the recommended option for FOSS users, as it is an easier way to save and visualize OpenVox Server metrics. The `grafanadash` module is still useful for users +exporting to their existing Graphite installation. + +> **Note:** The `grafanadash` and `puppet-graphite` modules referenced in this document are community modules, not OpenVox-supported. They are provided for testing and demonstration purposes _only_. + +## Getting started with Graphite + +[Graphite][] is a third-party monitoring application that stores real-time metrics and provides customizable ways to view them. OpenVox Server can export many metrics to Graphite, and exports a set of metrics +by default that is designed to be immediately useful to administrators. + +> **Note:** A Graphite setup is deeply customizable and can report many OpenVox Server metrics on demand. However, it requires considerable configuration and additional server resources. To retrieve metrics +> through HTTP requests, see the metrics API. + +To start using Graphite with OpenVox Server, you must: + +- [Install and configure a Graphite server](https://graphite.readthedocs.io/en/latest/install.html). +- [Enable OpenVox Server's Graphite support](#enabling-openvox-servers-graphite-support). + +[Grafana][] provides a web-based customizable dashboard that's compatible with Graphite, and the [`grafanadash`][] module installs and configures it by default. + +### Using the `grafanadash` module to quickly set up a Graphite demo server + +The [`grafanadash`][] module quickly installs and configures a basic test instance of [Graphite][] with the [Grafana][] extension. When installed on a dedicated agent, this module provides a quick +demonstration of how Graphite and Grafana can consume and display OpenVox Server metrics. + +> **WARNING:** The `grafanadash` module is _not_ an OpenVox-supported module. It is designed for testing and demonstration purposes _only_, and tested against CentOS 6 only. +> +> Also, install this module on a dedicated agent _only_. Do **not** install it on the node running OpenVox Server, because the module makes security policy changes that are inappropriate for a server: +> +> - SELinux can cause issues with Graphite and Grafana, so the module temporarily disables SELinux. If you reboot the machine after using the module to install Graphite, you must disable SELinux again and +> restart the Apache service to use Graphite and Grafana. +> - The module disables the `iptables` firewall and enables cross-origin resource sharing on Apache, which are potential security risks. + +#### Installing the `grafanadash` module + +Install the `grafanadash` module on a \*nix agent. The module's `grafanadash::dev` class installs and configures a Graphite server, the Grafana extension, and a default dashboard. + +1. Install a \*nix agent to serve as the Graphite server. + +2. As root on the agent node, run `puppet module install puppetlabs-grafanadash`. + +3. As root on the agent node, run `puppet apply -e 'include grafanadash::dev'`. + +#### Running Grafana + +Grafana runs as a web dashboard, and the `grafanadash` module configures it to use port 10000 by default. To view OpenVox Server metrics in Grafana, you must create a metrics dashboard, or edit and import a +JSON-based dashboard that includes OpenVox Server metrics, such as the [sample Grafana dashboard][] that we provide. + +1. In a web browser on a computer that can reach the agent node running Grafana, navigate to `http://:10000`. + + There, you'll see a test screen that indicates whether Grafana can successfully connect to your Graphite server. + + If Grafana is configured to use a hostname that the computer on which the browser is running cannot resolve, click **view details** and then the **Requests** tab to determine the hostname Grafana is trying + to use. Next, add the IP address and hostname to the computer's `/etc/hosts` file on Linux or OS X, or `C:\Windows\system32\drivers\etc\hosts` file on Windows. + +2. Download and edit our [sample Grafana dashboard][], `sample_metrics_dashboard.json`. + + a. Open the `sample_metrics_dashboard.json` file in a text editor on the same computer you're using to access Grafana. + + b. Throughout the file, replace our sample hostname of `master.example.com` with your OpenVox Server's hostname. (**Note:** This value **must** be used as the `metrics_server_id` setting, as configured + below.) + + c. Save the file. + +3. In the Grafana UI, click **search** (the folder icon), then **Import**, then **Browse**. + +4. Navigate to and select the edited JSON file. + +This loads a dashboard with nine graphs that display various metrics exported from OpenVox Server to the Graphite server. (For details, see +[Using the Grafana dashboard](#using-the-sample-grafana-dashboard).) However, these graphs will remain empty until you enable OpenVox Server's Graphite metrics. + +> Note: If you want to integrate OpenVox Server's Grafana exporting with your own infrastructure, use the `grafanadash` module. If you want visualization of metrics, use the +> `puppetlabs-puppet_metrics_dashboard` module. + +### Enabling OpenVox Server's Graphite support + +Configure OpenVox Server's [`metrics.conf`](./config_file_metrics.html) file to enable and use the Graphite server. + +1. Set the `enabled` parameter to true in `metrics.registries.puppetserver.reporters.graphite`: + + ```text + metrics: { + server-id: localhost + registries: { + puppetserver: { + ... + reporters: { + ... + # enable or disable Graphite metrics reporter + graphite: { + enabled: true + } + } + } + } + } + ``` + +2. Configure the Graphite host settings in `metrics.reporters.graphite`: + - **host:** The Graphite host's IP address as a string. + - **port:** The Graphite host's port number. + - **update-interval-seconds:** How frequently OpenVox Server should send metrics to Graphite. + +3. Verify that `metrics.registries.puppetserver.reporters.jmx.enabled` is not set to false. Its default setting is true. + +> **Tip:** In the Grafana UI, choose an appropriate time window from the drop-down menu. + +### Using the sample Grafana dashboard + +The [sample Grafana dashboard][] provides what we think is an interesting starting point. You can click on the title of any graph, and then click **edit** to tweak the graphs as you see fit. + +- **Active requests:** This graph serves as a "health check" for OpenVox Server. It shows a flat line that represents the number of CPUs you have in your system, a metric that indicates the total number of + HTTP requests actively being processed by the server at any moment in time, and a rolling average of the number of active requests. If the number of requests being processed exceeds the number of CPUs for + any significant length of time, your server might be receiving more requests than it can efficiently process. + +- **Request durations:** This graph breaks down the average response times for different types of requests made by agents. This indicates how expensive catalog and report requests are compared to the other + types of requests. It also provides a way to see changes in catalog compilation times when you modify your Puppet code. A sharp curve upward for all of the types of requests indicates an overloaded server, + and they should trend downward after reducing the load on the server. + +- **Request ratios:** This graph shows how many requests of each type that OpenVox Server has handled. Under normal circumstances, you should see about the same number of catalog, node, or report requests, + because these all happen one time per agent run. The number of file and file metadata requests correlate to how many remote file resources are in the agents' catalogs. + +- **Communications with OpenVoxDB:** This graph tracks the amount of time it takes OpenVox Server to send data and requests for common operations to, and receive responses from, OpenVoxDB. + +- **JRubies**: This graph tracks how many JRubies are in use, how many are free, the mean number of free JRubies, and the mean number of requested JRubies. + + If the number of free JRubies is often less than one, or the mean number of free JRubies is less than one, OpenVox Server is requesting and consuming more JRubies than are available. This overload reduces + OpenVox Server's performance. While this might simply be a symptom of an under-resourced server, it can also be caused by poorly optimized Puppet code or bottlenecks in the server's communications with + OpenVoxDB if it is in use. + + If catalog compilation times have increased but OpenVoxDB performance remains the same, examine your Puppet code for potentially unoptimized code. If OpenVoxDB communication times have increased, tune + OpenVoxDB for better performance or allocate more resources to it. + + If neither catalog compilation nor OpenVoxDB communication times are degraded, the OpenVox Server process might be under-resourced on your server. If you have available CPU time and memory, + [increase the number of JRuby instances](./tuning_guide.html) to allow it to allocate more JRubies. Otherwise, consider adding additional compilers to distribute the catalog compilation load. + +- **JRuby Timers**: This graph tracks several JRuby pool metrics. + - The borrow time represents the mean amount of time that OpenVox Server uses ("borrows") each JRuby from the pool. + + - The wait time represents the total amount of time that OpenVox Server waits for a free JRuby instance. + + - The lock held time represents the amount of time that OpenVox Server holds a lock on the pool, during which JRubies cannot be borrowed. + + - The lock wait time represents the amount of time that OpenVox Server waits to acquire a lock on the pool. + + These metrics help identify sources of potential JRuby allocation bottlenecks. + +- **Memory Usage**: This graph tracks how much heap and non-heap memory that OpenVox Server uses. + +- **Compilation:** This graph breaks catalog compilation down into various phases to show how expensive each phase is on the server. + +### Example Grafana dashboard excerpt + +The following example shows only the `targets` parameter of a dashboard to demonstrate the full names of OpenVox Server's exported Graphite metrics (assuming the OpenVox Server instance has a domain of +`master.example.com`) and a way to add targets directly to an exported Grafana dashboard's JSON content. + +```json +"panels": [ + { + "span": 4, + "editable": true, + "type": "graphite", + +... + + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.num-cpus,'num cpus')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.active-requests.count,'active requests')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.active-histo.mean,'average')" + } + ], + "aliasColors": {}, + "aliasYAxis": {}, + "title": "Active Requests" + } +] +``` + +See the [sample Grafana dashboard][] for a detailed example of how a Grafana dashboard accesses these exported Graphite metrics. + +## Available Graphite metrics + +The following HTTP and Puppet profiler metrics are available from OpenVox Server and can be added to your metrics reporting. Each metric is prefixed with `puppetlabs.`; for instance, the +Grafana dashboard file refers to the `num-cpus` metric as `puppetlabs..num-cpus`. + +Additionally, metrics might be suffixed by fields, such as `count` or `mean`, that return more specific data points. For instance, the `puppetlabs..compiler.mean` metric returns only the mean +length of time it takes OpenVox Server to compile a catalog. + +To aid with reference, metrics in the list below are segmented into three groups: + +- **Statistical metrics:** Metrics that have all eight of these statistical analysis fields, in addition to the top-level metric: + - `max`: Its maximum measured value. + + - `min`: Its minimum measured value. + + - `mean`: Its mean, or average, value. + + - `stddev`: Its standard deviation from the mean. + + - `count`: An incremental counter. + + - `p50`: The value of its 50th percentile, or median. + + - `p75`: The value of its 75th percentile. + + - `p95`: The value of its 95th percentile. + +- **Counters only:** Metrics that only count a value, or only have a `count` field. + +- **Other:** Metrics that have unique sets of available fields. + +> **Note:** OpenVox Server can export many, many metrics -- so many that enabling all of them at large installations can overwhelm Grafana servers. To avoid this, OpenVox Server exports only a subset of its +> available metrics by default. This default set is designed to report the most relevant metrics for administrators monitoring performance and stability. +> +> To add to the default list of exported metrics, see [Modifying OpenVox Server's exported metrics](#modifying-openvox-servers-exported-metrics). + +OpenVox Server exports each metric in the lists below by default. + +### Statistical metrics + +#### Compiler metrics + +- `puppetlabs..compiler`: The time spent compiling catalogs. This metric represents the sum of the `compiler.compile`, `static_compile`, `find_facts`, and `find_node` fields. + - `puppetlabs..compiler.compile`: The total time spent compiling dynamic (non-static) catalogs. + + To measure specific nodes and environments, see [Modifying OpenVox Server's exported metrics](#modifying-openvox-servers-exported-metrics). + + - `puppetlabs..compiler.find_facts`: The time spent parsing facts. + + - `puppetlabs..compiler.find_node`: The time spent retrieving node data. If the Node Classifier (or another ENC) is configured, this includes the time spent communicating with it. + + - `puppetlabs..compiler.static_compile`: The time spent compiling [static catalogs][]. + + - `puppetlabs..compiler.static_compile_inlining`: The time spent inlining metadata for static catalogs. + + - `puppetlabs..compiler.static_compile_postprocessing`: The time spent post-processing static catalogs. + +#### Function metrics + +- `puppetlabs..functions`: The amount of time during catalog compilation spent in function calls. The `functions` metric can also report any of the + [statistical metrics](#available-graphite-metrics) fields for a single function by specifying the function name as a field. + + For example, to report the mean time spent in a function call during catalog compilation, use `puppetlabs..functions..mean`. + +#### HTTP metrics + +- `puppetlabs..http.active-histo`: A histogram of active HTTP requests over time. + +- `puppetlabs..http.puppet-v3-catalog-/*/-requests`: The time OpenVox Server has spent handling catalog requests, including time spent waiting for an available JRuby instance. + +- `puppetlabs..http.puppet-v3-environment-/*/-requests`: The time OpenVox Server has spent handling environment requests, including time spent waiting for an available JRuby instance. + +- `puppetlabs..http.puppet-v3-environment_classes-/*/-requests`: The time spent handling requests to the + [`environment_classes` API endpoint](./puppet-api/v3/environment_classes.html), which the Node Classifier uses to refresh classes. + +- `puppetlabs..http.puppet-v3-environments-requests`: The time spent handling requests to the `environments` API endpoint. + +- The following metrics measure the time spent handling file-related API endpoints: + - `puppetlabs..http.puppet-v3-file_bucket_file-/*/-requests` + + - `puppetlabs..http.puppet-v3-file_content-/*/-requests` + + - `puppetlabs..http.puppet-v3-file_metadata-/*/-requests` + + - `puppetlabs..http.puppet-v3-file_metadatas-/*/-requests` + +- `puppetlabs..http.puppet-v3-node-/*/-requests`: The time spent handling node requests, which are sent to the Node Classifier. A bottleneck here might indicate an issue with the Node + Classifier or OpenVoxDB. + +- `puppetlabs..http.puppet-v3-report-/*/-requests`: The time spent handling report requests. A bottleneck here might indicate an issue with OpenVoxDB. + +- `puppetlabs..http.puppet-v3-static_file_content-/*/-requests`: The time spent handling requests to the + [`static_file_content` API endpoint](./puppet-api/v3/static_file_content.html) used by Direct Puppet with file sync. + +#### JRuby metrics + +OpenVox Server uses an embedded JRuby interpreter to execute Ruby code. By default, JRuby spawns parallel instances known as JRubies to execute Ruby code, which occurs during most OpenVox Server activities. +When `multithreaded` is set to `true`, a single JRuby is used instead to process a limited number of threads in parallel. For each of these metrics, they refer to JRuby instances by default and JRuby threads +in multithreaded mode. + +See [Tuning JRuby on OpenVox Server](./tuning_guide.html) for details on adjusting JRuby settings. + +- `puppetlabs..jruby.borrow-timer`: The time spent with a borrowed JRuby. + +- `puppetlabs..jruby.free-jrubies-histo`: A histogram of free JRubies over time. This metric's average value should greater than 1; if it isn't, [more JRubies](./tuning_guide.html) or + another compiler might be needed to keep up with requests. + +- `puppetlabs..jruby.lock-held-timer`: The time spent holding the JRuby lock. + +- `puppetlabs..jruby.lock-wait-timer`: The time spent waiting to acquire the JRuby lock. + +- `puppetlabs..jruby.requested-jrubies-histo`: A histogram of requested JRubies over time. This increases as the number of free JRubies, or the `free-jrubies-histo` metric, decreases, + which can suggest that the server's capacity is being depleted. + +- `puppetlabs..jruby.wait-timer`: The time spent waiting to borrow a JRuby. + +#### OpenVoxDB metrics + +The following metrics measure the time that OpenVox Server spends sending or receiving data from OpenVoxDB. The metric names use the `puppetdb` identifier for compatibility with existing tooling. + +- `puppetlabs..puppetdb.catalog.save` + +- `puppetlabs..puppetdb.command.submit` + +- `puppetlabs..puppetdb.facts.find` + +- `puppetlabs..puppetdb.facts.search` + +- `puppetlabs..puppetdb.report.process` + +- `puppetlabs..puppetdb.resource.search` + +### Counters only + +#### HTTP metrics + +- `puppetlabs..http.active-requests`: The number of active HTTP requests. + +- The following counter metrics report the percentage of each HTTP API endpoint's share of total handled HTTP requests. + - `puppetlabs..http.puppet-v3-catalog-/*/-percentage` + + - `puppetlabs..http.puppet-v3-environment-/*/-percentage` + + - `puppetlabs..http.puppet-v3-environment_classes-/*/-percentage` + + - `puppetlabs..http.puppet-v3-environments-percentage` + + - `puppetlabs..http.puppet-v3-file_bucket_file-/*/-percentage` + + - `puppetlabs..http.puppet-v3-file_content-/*/-percentage` + + - `puppetlabs..http.puppet-v3-file_metadata-/*/-percentage` + + - `puppetlabs..http.puppet-v3-file_metadatas-/*/-percentage` + + - `puppetlabs..http.puppet-v3-node-/*/-percentage` + + - `puppetlabs..http.puppet-v3-report-/*/-percentage` + + - `puppetlabs..http.puppet-v3-resource_type-/*/-percentage` + + - `puppetlabs..http.puppet-v3-resource_types-/*/-percentage` + + - `puppetlabs..http.puppet-v3-static_file_content-/*/-percentage` + + - `puppetlabs..http.puppet-v3-status-/*/-percentage` + +- `puppetlabs..http.total-requests`: The total requests handled by OpenVox Server. + +#### JRuby metrics + +> **Note:** In multithreaded mode, each of these refers to JRuby threads instead of separate JRuby instances. + +- `puppetlabs..jruby.borrow-count`: The number of successfully borrowed JRubies. + +- `puppetlabs..jruby.borrow-retry-count`: The number of attempts to borrow a JRuby that must be retried. + +- `puppetlabs..jruby.borrow-timeout-count`: The number of attempts to borrow a JRuby that resulted in a timeout. + +- `puppetlabs..jruby.request-count`: The number of requested JRubies. + +- `puppetlabs..jruby.return-count`: The number of JRubies successfully returned to the pool. + +- `puppetlabs..jruby.num-free-jrubies`: The number of free JRuby instances. If this number is often 0, more requests are coming in than the server has available JRuby instances. To + alleviate this, increase the number of JRuby instances on the server or add additional compilers. + +- `puppetlabs..jruby.num-jrubies`: The total number of JRuby instances on the server, governed by the `max-active-instances` setting. See + [Tuning JRuby on OpenVox Server](./tuning_guide.html) for details. + +### Other metrics + +These metrics measure raw resource availability and capacity. + +- `puppetlabs..num-cpus`: The number of available CPUs on the server. + +- `puppetlabs..uptime`: The OpenVox Server process's uptime. + +- Total, heap, and non-heap memory that's committed (`committed`), initialized (`init`), and used (`used`), and the maximum amount of memory that can be used (`max`). + - `puppetlabs..memory.total.committed` + + - `puppetlabs..memory.total.init` + + - `puppetlabs..memory.total.used` + + - `puppetlabs..memory.total.max` + + - `puppetlabs..memory.heap.committed` + + - `puppetlabs..memory.heap.init` + + - `puppetlabs..memory.heap.used` + + - `puppetlabs..memory.heap.max` + + - `puppetlabs..memory.non-heap.committed` + + - `puppetlabs..memory.non-heap.init` + + - `puppetlabs..memory.non-heap.used` + + - `puppetlabs..memory.non-heap.max` + +For details about HTTP client metrics, which measure performance of OpenVox Server's requests to other services, see [their documentation][HTTP client metrics]. + +### Modifying OpenVox Server's exported metrics + +In addition to the above default metrics, you can also export metrics measuring specific environments and nodes. + +The `metrics.registries.puppetserver.metrics-allowed` parameter in [`metrics.conf`][] takes an array of strings representing the metrics you want to enable. + +Omit the `puppetlabs.` prefix and field suffixes (such as `.count` or `.mean`) from metrics when adding them to this class. Instead, suffix the environment or node name as a field to the +metric. + +For example, to track the compilation time for the `production` environment, add `compiler.compile.production` to the `metrics-allowed` list. To track only the `my.node.localdomain` node in the `production` +environment, add `compiler.compile.production.my.node.localdomain` to the `metrics-allowed` list. + +Optional metrics include: + +- `compiler.compile.` and `compiler.compile..`, and all statistical fields suffixed to these (such as `compiler.compile..mean`). + +- `compiler.compile.evaluate_resources.`: Time spent evaluating a specific resource during catalog compilation. diff --git a/docs/_openvox-server_9x/puppet_server_metrics_performance.markdown b/docs/_openvox-server_9x/puppet_server_metrics_performance.markdown new file mode 100644 index 000000000..73fcc5c60 --- /dev/null +++ b/docs/_openvox-server_9x/puppet_server_metrics_performance.markdown @@ -0,0 +1,122 @@ +--- +layout: default +title: "Applying metrics to improve performance" +--- + +[metrics]: ./puppet_server_metrics.html +[tuning guide]: ./tuning_guide.html +[status API]: ./status-api/v1/services.html +[sample Grafana dashboard]: ./sample-puppetserver-metrics-dashboard.json +[puppetserver.conf]: ./config_file_puppetserver.html +[HTTP client metrics]: ./http_client_metrics.html + +OpenVox Server produces [several types of metrics][metrics] that administrators can use to identify performance bottlenecks or capacity issues. Interpreting this data is largely up to you and depends on many +factors unique to your installation and usage, but there are some common trends in metrics that you can use to make OpenVox Server function better. + +> **Note:** This document assumes that you are already familiar with OpenVox Server's [metrics tools][metrics], which report on relevant information, and its [tuning guide][], which provides instructions for +> modifying relevant settings. To put it another way, this guide attempts to explain questions about "why" OpenVox Server performs the way it does for you, while your servers are the "who", Server [metrics][] +> help you track down exactly "what" is affecting performance, and the [tuning guide][] explains "how" you can improve performance. +> +## Measuring capacity with JRubies + +OpenVox Server uses JRuby, which rations server resources in the form of JRuby instances in default mode, and JRuby threads in multithreaded mode. OpenVox Server consumes these as it handles requests. A simple +way of explaining OpenVox Server performance is to remember that your Server infrastructure must be capable of providing enough JRuby instances or threads for the amount of activity it handles. Anything that +reduces or limits your server's capacity to produce JRubies also degrades OpenVox Server's performance. + +Several factors can limit your Server infrastructure's ability to produce JRubies. + +### Request-handling capacity + +> **Note:** These guidelines for interpreting metrics generally apply to both default and multithreaded mode. However, threads are much cheaper in terms of system resources, since they do not need to duplicate +> all of Puppet's runtime, so you may have more vertical scalability in multithreaded mode. + +If your free JRubies are 0 or fewer, your server is receiving more requests for JRubies than it can provide, which means it must queue those requests to wait until resources are available. OpenVox Server +performs best when the average number of free JRubies is above 1, which means Server always has enough resources to immediately handle incoming requests. + +There are two indicators in OpenVox Server's metrics that can help you identify a request-handling capacity issue: + +- **Average JRuby Wait Time:** This refers to the amount of time OpenVox Server has to wait for an available JRuby to become available, and increases when each JRuby is held for a longer period of time, which + reduces the overall number of free JRubies and forces new requests to wait longer for available resources. +- **Average JRuby Borrow Time:** This refers to the amount of time that OpenVox Server "holds" a JRuby as a resource for a request, and increases because of other factors on the server. + +If wait time increases but borrow time stays the same, your Server infrastructure might be serving too many agents. This indicates that Server can easily handle requests but is receiving too many at one time to +keep up. + +If both wait and borrow times are increasing, something else on your server is causing requests to take longer to process. The longer borrow times suggest that OpenVox Server is struggling more than before to +process requests, which has a cascading effect on wait times. Correlate borrow time increases with other events whenever possible to isolate what activities might cause them, such as a Puppet code change. + +If you are setting up OpenVox Server for the first time, start by increasing your Server infrastructure's capacity through additional JRubies (if your server has spare CPU and memory resources) or compilers +until you have more than 0 free JRubies, and your average number of free JRubies are at least 1. After your system can handle its request volume, you can start looking into more specific performance +improvements. + +#### Adding more JRubies + +If you must add JRubies, remember that OpenVox Server is tuned by default to use one fewer than your total number of CPUs, with a maximum of 4 CPUs, for the number of available JRubies. You can change this by +setting `max-active-instances` in [`puppetserver.conf`][puppetserver.conf], under the `jruby-puppet` section. In the default mode, increasing `max-active-instances` creates whole independent JRuby instances. In +multithreaded mode, this setting instead controls the number of threads that the single JRuby instance will process concurrently, and therefore has different scaling characteristics. Tuning recommendations for +this mode are under development. + +When running in the default mode, follow these guidelines for allocating resources when adding JRubies: + +Each JRuby also has a certain amount of persistent memory overhead required in order to load both Puppet's Ruby code and your Puppet code. In other words, your available memory sets a baseline limit to how much +Puppet code you can process. Catalog compilation can consume more memory, and OpenVox Server's total memory usage depends on the number of agents being served, how frequently those agents check in, how many +resources are being managed on each agent, and the complexity of the manifests and modules in use. + +With the `jruby-puppet.compile-mode` setting in [`puppetserver.conf`][puppetserver.conf] set to `off`, a JRuby requires at least 40MB of memory under JRuby 1.7 and at least 60MB under JRuby9k in order to +compile a nearly empty catalog. This includes memory for the scripting container, Puppet's Ruby code and additional memory overhead. + +For real-world catalogs, you can generally add an absolute minimum of 15MB for each additional JRuby, based on comparing a minimal catalog +compilation to compiling a catalog for a basic role that installs Tomcat and Postgres servers. + +Your Puppet-managed infrastructure is probably larger and more complex than that test scenario, and every complication adds more to each additional JRuby's memory requirements. (For instance, we recommend +assuming that OpenVox Server will use [at least 512MB per JRuby](tuning_guide.html) while under load.) You can calculate a similar value unique to your +infrastructure by measuring `puppetserver` memory usage during your infrastructure's catalog compilations and comparing it to compiling a minimal catalog for a similar number of nodes. + +The `jruby-metrics` section of the [status API][] endpoint also lists the `requested-instances`, which shows what requests have come in that are waiting to borrow a JRuby instance. This part of the status +endpoint lists the lock's status, how many times it has been requested, and how long it has been held for. If it is currently being held and has been held for a while, you might see requests starting to stack +up in the `requested-instances` section. + +#### Adding compilers + +If you don't have the additional capacity on your server to add more JRubies, you'll want to add another compiler to your Server infrastructure. See +[Scaling OpenVox Server with compilers](./scaling_puppet_server.html). + +### HTTP request delays + +If JRuby metrics appear to be stable, performance issues might originate from lag in server requests, which also have a cascading effect on other metrics. HTTP metrics in the [status API][], and the requests +graph in the [Grafana dashboard](./puppet_server_metrics.html), can help you determine when and where request times have increased. + +HTTP metrics include the total time for the server to handle the request, including waiting for a JRuby instance to become available. When JRuby borrow time increases, wait time also increases, so when borrow +time for _one_ type of request increases, wait times for _all_ requests increases. + +Catalog compilation, which is graphed on the [sample Grafana dashboard][], most commonly increases request times, because there are many points of potential failure or delay in a catalog compilation. Several +things could cause catalog compilation lengthen JRuby borrow times. + +- A Puppet code change, such as a faulty or complex new function. The Grafana dashboard should show if functions start taking significantly longer, and the experimental dashboard and [status API][] endpoint + also list the lengthiest function calls (showing the top 10 and top 40, respectively) based on aggregate execution times. +- Adding many file resources at one time. + +In cases like these, there might be more efficient ways to author your Puppet code, you might be extending Puppet to the point where you need to add JRubies or compilers even if you aren't adding more +agents. + +Slowdowns in OpenVoxDB can also cause catalog compilations to take more time: if you use exported resources or the `puppetdb_query` function and OpenVoxDB has a problem, catalog compilation times will increase. + +OpenVox Server also sends agents' facts and the compiled catalog to OpenVoxDB during catalog compilation. The [status API][] for the master service reports metrics for these operations under +[`http-client-metrics`][HTTP client metrics], and in the Grafana dashboard in the "External HTTP Communications" graph. + +OpenVox Server also requests facts as HTTP requests while handling a node request, and submits reports via HTTP requests while handling of a report request. If you have an HTTP report processor set up, the +Grafana dashboard shows metrics for `Http report processor,` as does the [status API][] endpoint under `http-client-metrics` in the master service, for metric ID `['puppet', 'report', 'http']`. Delays in the +report processor are passed on to OpenVox Server. + +### Memory leaks and usage + +A memory leak or increased memory pressure can stress OpenVox Server's available resources. In this case, the Java VM will spend more time doing garbage collection, causing the GC time and GC CPU % metrics to +increase. These metrics are available from the [status API][] endpoint, as well as in the mbeans metrics available from both the [`/metrics/v1/mbeans`](./metrics-api/v1/metrics_api.html) or +[`/metrics/v2/`](./metrics-api/v2/metrics_api.html) endpoints. + +If you can't identify the source of a memory leak, setting the `max-requests-per-instance` setting in [`puppetserver.conf`][puppetserver.conf] to something other than the default of 0 limits the number of +requests a JRuby handles during its lifetime and enables automatic JRuby flushing. Enabling this setting reduces overall performance, but if you enable it and no longer see signs of persistent memory leaks, +check your module code for inefficiencies or memory-consuming bugs. + +> **Note:** In multithreaded mode, the `max-requests-per-instance` setting refers to the sum total number of requests processed by the single JRuby instance, across all of its threads. While that single JRuby +> is being flushed, all requests will suspend until the instance becomes available again. diff --git a/docs/_openvox-server_9x/release_notes.markdown b/docs/_openvox-server_9x/release_notes.markdown new file mode 100644 index 000000000..0837747de --- /dev/null +++ b/docs/_openvox-server_9x/release_notes.markdown @@ -0,0 +1,94 @@ +--- +layout: default +title: "OpenVox Server: Release Notes" +--- + +> **OpenVox Server 9 is in prerelease.** This page tracks the beta builds ahead of the +> stable 9.0.0 release. Expect breaking changes between prereleases; see +> [known issues](known_issues.html) for anything discovered so far. + +OpenVox Server 9 pairs with OpenVox 9 (the agent): the `openvox-server` 9.x package +depends on `openvox-agent` 9.x on the same host. For the changes on the agent side, +see the [OpenVox 9 release notes](/openvox/9.x/release_notes.html). + +## OpenVox Server 9.0.0-beta5 + +Released August 12, 2026. + +This is a **prerelease** of OpenVox Server 9 and is not yet the stable release. See the +[project's GitHub release page](https://github.com/OpenVoxProject/openvox-server/releases/tag/9.0.0-beta5) +for the full list of changes. + +Notable changes in this build: + +- Requests that include the system trust store (for example, report processors posting to + external HTTPS endpoints) fall back to the PEM CA bundle shipped with the OpenVox 9 + agent. The 9.x agent runtime no longer ships the Java keystore the earlier betas relied + on, which caused `PKIX path building failed` errors when connecting to publicly signed + endpoints. + +## OpenVox Server 9.0.0-beta4 + +Released August 6, 2026. + +This build republishes the 9.0.0-beta3 changes. The 9.0.0-beta3 packages were never +published because of a CI problem, and a `9.0.0` artifact containing the beta3 changes +was published to Clojars by mistake. See the +[project's GitHub release page](https://github.com/OpenVoxProject/openvox-server/releases/tag/9.0.0-beta4). + +## OpenVox Server 9.0.0-beta3 + +Released August 6, 2026. Packages for this build were not published; use 9.0.0-beta4. + +This is a **prerelease** of OpenVox Server 9 and is not yet the stable release. It includes +breaking changes; see the +[project's GitHub release page](https://github.com/OpenVoxProject/openvox-server/releases/tag/9.0.0-beta3) +for the full list of changes. + +Notable breaking changes in this build: + +- Reading content back out of the filebucket (`GET` and `POST` on + `/puppet/v3/file_bucket_file`) now requires a client certificate with the + `pp_cli_auth: "true"` extension in the default [`auth.conf`](./config_file_auth.html). + Agents keep `HEAD` and `PUT`, which is all they need to store backups. +- The `gettext` gem is no longer vendored with the JRuby gems. +- The package now requires `openvox-agent` 9.0.0-beta2 or later. + +## OpenVox Server 9.0.0-beta2 + +Released July 27, 2026. + +This is a **prerelease** of OpenVox Server 9 and is not yet the stable release. It includes +breaking changes; see the +[project's GitHub release page](https://github.com/OpenVoxProject/openvox-server/releases/tag/9.0.0-beta2) +for the full list of changes. + +Notable changes in this build: + +- The package now depends on `openvox-agent` 9.0.0-beta1 or later. +- JRuby is upgraded to 10.1.1.0, which targets Ruby 4.0 compatibility (the previous 10.0.x + targeted Ruby 3.4). +- `gem install` works again during FIPS builds. +- Fixed the `resolv` regression when querying IPv6 DNS servers. + +## OpenVox Server 9.0.0-beta1 + +Released July 15, 2026. + +This is the first **prerelease** of OpenVox Server 9 and is not yet the stable release. It +includes breaking changes; see the +[project's GitHub release page](https://github.com/OpenVoxProject/openvox-server/releases/tag/9.0.0-beta1) +for the full list of changes. + +Notable breaking changes in this build: + +- Java 21 or 25 is required. Java 17 is no longer supported. +- The embedded web server is upgraded to Jetty 12. +- JRuby is upgraded to the 10.x series. +- The `pe_serverversion` fact is removed. +- Packages are no longer built for Debian 11, Debian 12, or Amazon Linux 2. + +Other notable changes: + +- Fixed the Jolokia 2.x configuration for the [v2 metrics API](./metrics-api/v2/metrics_api.html). +- OpenVox Server now reports service readiness to Trapperkeeper. diff --git a/docs/_openvox-server_9x/restarting.markdown b/docs/_openvox-server_9x/restarting.markdown new file mode 100644 index 000000000..04ba5b2df --- /dev/null +++ b/docs/_openvox-server_9x/restarting.markdown @@ -0,0 +1,80 @@ +--- +layout: default +title: "Puppet Server: Restarting the Server" +canonical: "/puppetserver/latest/restarting.html" +--- + +[logback.xml]: ./config_file_logbackxml.html +[Hiera]: /openvox/latest/hiera_intro.html +[gems]: ./gems.html +[core dependencies]: /openvox/latest/about_agent.html#what-openvox-agent-provides +[environment]: /openvox/latest/environments_about.html +[environment caching]: /openvox/latest/configuration.html#environment_timeout + +Starting in version 2.3.0, you can restart Puppet Server by sending a hangup signal, also known as a [HUP signal or SIGHUP](https://en.wikipedia.org/wiki/SIGHUP), to the running Puppet Server process. The HUP +signal stops Puppet Server and reloads it gracefully, without terminating the JVM process. This is generally _much_ faster than completely stopping and restarting the process. This allows you to quickly load +changes to your Puppet Server master, including configuration changes. + +There are several ways to send a HUP signal to the Puppet Server process, but the most straightforward is to run the following [`kill`](http://linux.die.net/man/1/kill) command: + + kill -HUP `pgrep -f puppet-server` + +Starting in version 2.7.0, you can also reload Puppet Server by running the "reload" action via the operating system's service framework. This is analogous to sending a hangup signal but with the benefit of +having the "reload" command pause until the server has been completely reloaded, similar to how the "restart" command pauses until the service process has been fully restarted. Advantages to using the "reload" +action as opposed to just sending a HUP signal include: + +1. Unlike with the HUP signal approach, you do not have to determine the process ID of the puppetserver process to be reloaded. + +2. When using the HUP signal with an automated script (or Puppet code), it is possible that any additional commands in the script might behave improperly if performed while the server is still reloading. With + the "reload" command, though, the server should be up and using its latest configuration before any subsequent script commands are performed. + +3. Even if the server fails to reload and shuts down --- for example, due to a configuration error --- the `kill -HUP` command might still return a 0 (success) exit code. With the "reload" command, however, any + configuration change which causes the server to shut down will produce a non-0 (failure) exit code. The "reload" command, therefore, would allow you to more reliably determine if the server failed to reload + properly. + +Use the following commands to perform the "reload" action for Puppet Server. + +All current OS distributions: + + service puppetserver reload + +OS distributions which use sysvinit-style scripts: + + /etc/init.d/puppetserver reload + +OS distributions which use systemd service configurations: + + systemctl reload puppetserver + +## Restarting Puppet Server to pick up changes + +There are three ways to trigger your Puppet Server environment to refresh and pick up changes you've made. A request to the [HTTP Admin API to flush the JRuby pool](./admin-api/v1/jruby-pool.html) is the +quickest, but picks up only certain types of changes. A HUP signal or service reload is also quick, and applies additional changes. Other changes require a full Puppet Server restart. + +> **Note:** Changes to Puppet Server's [logging configuration in `logback.xml`][logback.xml] don't require a server restart. Puppet Server recognizes and applies them automatically, though it can take a minute +> or so for this to happen. However, you can restart the service to force it to recognize those changes. + +### Changes applied after a JRuby pool flush, HUP signal, service reload, or full Server restart + +- Changes to your `hiera.yaml` file to change your [Hiera][] configuration. +- [Installation or removal of gems][gems] for Puppet Server by `puppetserver gem`. +- Changes to the Ruby code for Puppet's [core dependencies][], such as Puppet, Facter, and Hiera. +- Changes to Puppet modules in an [environment][] where you've enabled [environment caching][]. You can also achieve this by making a request to the + [Admin API endpoint for flushing the environment cache](./admin-api/v1/environment-cache.html). +- Changes to the CA CRL file. For example, a `puppetserver ca clean` + +### Changes applied after a HUP signal, service reload, or full Server restart + +- Changes to Puppet Server [configuration files](./configuration.html) in its `conf.d` directory. +- Changes to the CA CRL file. For example, a `puppetserver ca clean` + +### Changes that require a full Server restart + +- Changes to JVM arguments, such as [heap size settings](./tuning_guide.html#jvm-heap-size), that are typically configured in your `/etc/sysconfig/puppetserver` or `/etc/default/puppetserver` file. +- Changes to [`ca.cfg`](./configuration.html#service-bootstrapping) to enable or disable Puppet Server's certificate authority (CA) service. + +For these types of changes, you must restart the process by using the operating system's service framework, for example, by using the `systemctl` or `service` commands. + +> Note: To ensure that the Puppet master and CA service is running in a platform agnostic way, use the `puppet resource service puppetserver ensure=running` command. This command is equivalent to +> `systemctl start puppetserver` on systems that support it. For more information on the resource command and managing a server’s desired state, see +> [Man Page: puppet resource](/openvox/latest/man/resource.html) and [Resource Type: service](/openvox/latest/types/service.html). diff --git a/docs/_openvox-server_9x/sample-puppetserver-metrics-dashboard.json b/docs/_openvox-server_9x/sample-puppetserver-metrics-dashboard.json new file mode 100644 index 000000000..3cfe6e3e6 --- /dev/null +++ b/docs/_openvox-server_9x/sample-puppetserver-metrics-dashboard.json @@ -0,0 +1,758 @@ +{ + "title": "Puppet Server", + "services": { + "filter": { + "list": [], + "time": { + "from": "now-60m", + "to": "now" + } + } + }, + "rows": [ + { + "collapse": false, + "height": "300px", + "panels": [ + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "* `num jrubies`: number of JRuby instances (constant)\n* `num free jrubies`: number of JRuby instances free\n* `average free jrubies`: average free JRubies over time\n* `average requested jrubies`: average requested JRubies over time", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": 0 + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.jruby.num-jrubies,'num jrubies')" + }, + { + "target": "alias(puppetlabs.master.example.com.jruby.num-free-jrubies,'num free jrubies')" + }, + { + "target": "alias(puppetlabs.master.example.com.jruby.free-jrubies-histo.mean,'average free jrubies')" + }, + { + "target": "alias(puppetlabs.master.example.com.jruby.requested-jrubies-histo.mean,'average requested jrubies')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "JRuby Usage", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "short", + "show": true + }, + { + "format": "short", + "show": true + } + ], + "zerofill": true + }, + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "* `borrow time`: average time a JRuby is borrowed for\n* `wait time`: average time Puppet Server spends waiting to borrow a JRuby to service a request\n* `lock held time`: average time the lock on the JRuby pool is held\n* `lock wait time`: average time Puppet Server spends waiting to acquire the lock", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.jruby.borrow-timer.mean,'borrow time')" + }, + { + "target": "alias(puppetlabs.master.example.com.jruby.wait-timer.mean,'wait time')" + }, + { + "target": "alias(puppetlabs.master.example.com.jruby.lock-held-timer.mean,'lock held time')" + }, + { + "target": "alias(puppetlabs.master.example.com.jruby.lock-wait-timer.mean,'lock wait time')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "JRuby Pool Timers", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "ms", + "show": true + }, + { + "format": "short", + "show": true + } + ], + "zerofill": true + }, + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "Heap and non-heap memory used, number of cpus available.", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [ + { + "alias": "num cpus", + "yaxis": 2 + } + ], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.memory.heap.used,'heap used')" + }, + { + "target": "alias(puppetlabs.master.example.com.memory.non-heap.used,'non-heap used')" + }, + { + "target": "alias(puppetlabs.master.example.com.num-cpus, 'num cpus')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "JVM metrics", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "bytes", + "label": "Memory", + "min": null, + "show": true + }, + { + "format": "short", + "label": "CPUs", + "logBase": 1, + "min": "0", + "show": true + } + ], + "zerofill": true + } + ], + "repeat": null, + "repeatIteration": null, + "repeatRowId": null + }, + { + "collapse": false, + "height": "300px", + "panels": [ + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "Average request time by type - time the server spends serving each type of request, including time waiting to acquire a JRuby", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.http.puppet-v3-catalog-.*.-requests.mean,'catalog')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.puppet-v3-report-.*.-requests.mean,'report')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.puppet-v3-node-.*.-requests.mean,'node')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.puppet-v3-file_metadatas-.*.-requests.mean,'file_metadatas')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.puppet-v3-file_metadata-.*.-requests.mean,'file_metadata')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.puppet-v3-file_content-.*.-requests.mean,'file_content')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.other-requests.mean,'other')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.total-requests.mean,'all')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Request Durations", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "ms", + "show": true + }, + { + "format": "short", + "show": true + } + ], + "zerofill": true + }, + { + "aliasColors": { + "create scope": "#BADFF4" + }, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "Time spent in different parts of handling a Catalog request\n\n* `compile`: time to compile catalogs.\n* `static_compile`: time to do a static compile of catalogs.\n* `static compile postprocessing`: time to do postprocessing of static catalog, including inlining file metadata.\n* `function calls`: time spent calling functions (included in `compile` time)\n* `find_node`: time to get node object (not included in `compile` time)\n* `find_facts`: time to parse facts sent as part of catalog request (not included in `compile` time)", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.compiler.compile.mean,'compile')" + }, + { + "target": "alias(puppetlabs.master.example.com.functions.mean,'function calls')" + }, + { + "target": "alias(puppetlabs.master.example.com.compiler.find_node.mean,'find_node')" + }, + { + "target": "alias(puppetlabs.master.example.com.compiler.static_compile.mean,'static compile')" + }, + { + "target": "alias(puppetlabs.master.example.com.compiler.static_compile_postprocessing.mean,'static compile postprocessing')" + }, + { + "target": "alias(puppetlabs.master.example.com.compiler.find_facts.mean,'find_facts')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Compilation", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "ms", + "show": true + }, + { + "format": "short", + "show": true + } + ], + "zerofill": true + }, + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "Time to make HTTP requests to other services\n\n* `PuppetDB: replace facts`: time to send a \"facts replace\" command to PuppetDB (part of a catalog compile)\n* `PuppetDB: replace catalog`: time to send a \"catalog replace\" command to PuppetDB (part of a catalog compile)\n* `PuppetDB: store report`: time to send a \"report store\" command to PuppetDB (part of handling a report request)\n* `PuppetDB: find facts`: time to get facts from PuppetDB (part of a node and catalog request, if the facts for the node are not cached)\n* `PuppetDB: search resources`: time to get exported resources from PuppetDB (part of a catalog compile if exported resources are used)\n*`PuppetDB: query`: time to execute a PuppetDB query (part of a catalog compile if the puppetdb query function is used)\n* `Http report processor`: time to submit a report to the http report url (if the http report processor is used)", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppetdb.command.replace_facts.full-response.mean, 'PuppetDB: replace facts')" + }, + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppetdb.command.replace_catalog.full-response.mean, 'PuppetDB: replace catalog')" + }, + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppetdb.command.store_report.full-response.mean, 'PuppetDB: store report')" + }, + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppetdb.facts.find.full-response.mean, 'PuppetDB: find facts')" + }, + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppetdb.resource.search.full-response.mean, 'PuppetDB: search resources')" + }, + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppetdb.query.full-response.mean, 'PuppetDB: query')" + }, + { + "target": "alias(puppetlabs.master.example.com.http-client.experimental.with-metric-id.puppet.report.http.full-response.mean, 'Http report processor')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "External HTTP Communications", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "ms", + "show": true + }, + { + "format": "short", + "show": true + } + ], + "zerofill": true + } + ], + "repeat": null, + "repeatIteration": null, + "repeatRowId": null + }, + { + "collapse": false, + "height": "300px", + "panels": [ + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "Percentage of requests handled of each type.", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": true, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(scale(puppetlabs.master.example.com.http.puppet-v3-catalog-.*.-percentage,100),'catalog')" + }, + { + "target": "alias(scale(puppetlabs.master.example.com.http.puppet-v3-report-.*.-percentage,100),'report')" + }, + { + "target": "alias(scale(puppetlabs.master.example.com.http.puppet-v3-node-.*.-percentage,100),'node')" + }, + { + "target": "alias(scale(puppetlabs.master.example.com.http.puppet-v3-file_metadatas-.*.-percentage,100),'file_metadatas')" + }, + { + "target": "alias(scale(puppetlabs.master.example.com.http.puppet-v3-file_metadata-.*.-percentage,100),'file_metadata')" + }, + { + "target": "alias(scale(puppetlabs.master.example.com.http.puppet-v3-file_content-.*.-percentage,100),'file_content')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Request Ratios", + "tooltip": { + "query_as_alias": true, + "value_type": "individual" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "short", + "label": "Percentage of Requests", + "show": true + }, + { + "format": "short", + "label": "", + "show": true + } + ], + "zerofill": true + }, + { + "aliasColors": {}, + "annotate": { + "enable": false + }, + "bars": false, + "datasource": null, + "description": "Number of active requests and average number of active requests (over ~ 5 minutes).", + "editable": true, + "fill": 0, + "grid": { + "max": null, + "min": null + }, + "legend": { + "avg": false, + "current": false, + "max": false, + "min": false, + "show": true, + "total": false, + "values": false + }, + "lines": true, + "linewidth": 1, + "loadingEditor": false, + "nullPointMode": "connected", + "percentage": false, + "pointradius": 5, + "points": false, + "renderer": "flot", + "resolution": 100, + "scale": 1, + "seriesOverrides": [], + "spaceLength": 10, + "span": 4, + "stack": false, + "steppedLine": false, + "targets": [ + { + "target": "alias(puppetlabs.master.example.com.http.active-requests.count,'active requests')" + }, + { + "target": "alias(puppetlabs.master.example.com.http.active-histo.mean, 'average number of active requests')" + } + ], + "thresholds": [], + "timeFrom": null, + "timeShift": null, + "title": "Active Requests", + "tooltip": { + "query_as_alias": true, + "value_type": "cumulative" + }, + "type": "graphite", + "xaxis": { + "buckets": null, + "mode": "time", + "name": null, + "show": true, + "values": [] + }, + "yaxes": [ + { + "format": "short", + "show": true + }, + { + "format": "short", + "show": true + } + ], + "zerofill": true + } + ], + "repeat": null, + "repeatIteration": null, + "repeatRowId": null + } + ], + "editable": true, + "failover": false, + "time": { + "from": "now-30m", + "to": "now" + }, + "timepicker": { + "collapse": false, + "enable": true, + "notice": false, + "now": true, + "refresh_intervals": [ + "5s", + "10s", + "30s", + "1m", + "5m", + "15m", + "30m", + "1h", + "2h", + "1d" + ], + "status": "Stable", + "time_options": [ + "5m", + "15m", + "1h", + "6h", + "12h", + "24h", + "2d", + "7d", + "30d" + ], + "type": "timepicker" + }, + "refresh": "30s", + "timezone": "browser" +} diff --git a/docs/_openvox-server_9x/scaling_puppet_server.markdown b/docs/_openvox-server_9x/scaling_puppet_server.markdown new file mode 100644 index 000000000..40cdf6674 --- /dev/null +++ b/docs/_openvox-server_9x/scaling_puppet_server.markdown @@ -0,0 +1,173 @@ +--- +layout: default +title: "Scaling OpenVox Server with compilers" +--- + +To scale OpenVox Server for many thousands of nodes, you'll need to add servers dedicated to catalog compilation. +These are known as **compilers**, and are simply additional load-balanced OpenVox Servers that independently +compile catalogs for agents from a shared codebase. + +For a reference architecture covering the full load-balanced stack (HAProxy, r10k, Foreman, and automation), +see [Load balanced architecture](https://voxpupuli.org/docs/arch_load_balanced) on the Vox Pupuli site. + +## Planning your load-balancing strategy + +The rest of your configuration depends on how you plan on distributing the agent load. Determine what your +deployment will look like before you add any compilers, but **implement load balancing as the last step** only +after you have the infrastructure in place to support it. + +### Using round-robin DNS + +Leave all of your agents pointed at the same OpenVox Server hostname, then configure your site's DNS to +arbitrarily route all requests directed at that hostname to the pool of available servers. + +For instance, if all of your agent nodes are configured with `server = puppet.example.com`, configure a DNS name such as: + +```text +# IP address of server 1: +puppet.example.com. IN A 192.0.2.50 +# IP address of server 2: +puppet.example.com. IN A 198.51.100.215 +``` + +For this option, configure your servers with `dns_alt_names` before their certificate request is made. + +### Using a hardware load balancer + +You can also use a hardware load balancer or a load-balancing proxy webserver to redirect requests more +intelligently. Depending on your configuration (for instance, SSL using either raw TCP proxying or acting as +its own SSL endpoint), you might also need to use other procedures in this document. + +Configuring a load balancer depends on the product, and is beyond the scope of this document. + +### Using DNS `SRV` Records + +You can use DNS `SRV` records to assign a pool of OpenVox Servers for agents to communicate with. This requires +a DNS service capable of `SRV` records, which includes all major DNS software. + +> **Note:** This method makes a large number of DNS requests. Request timeouts are completely under the DNS +> server's control and agents cannot cancel requests early. SRV records don't interact well with static servers +> set in the config file. Please keep these potential pitfalls in mind when configuring your DNS! + +Configure each of your agents with a `srv_domain` instead of a `server` in `puppet.conf`: + +```text +[main] +use_srv_records = true +srv_domain = example.com +``` + +Agents will then look up a `SRV` record at `_x-puppet._tcp.example.com` when they need to talk to an OpenVox Server. + +```text +# Equal-weight load balancing between server-a and server-b: +_x-puppet._tcp.example.com. IN SRV 0 5 8140 server-a.example.com. +_x-puppet._tcp.example.com. IN SRV 0 5 8140 server-b.example.com. +``` + +You can also implement more complex configurations. For instance, if all devices in site A are configured with +a `srv_domain` of `site-a.example.com`, and all nodes in site B are configured to `site-b.example.com`, you +can configure them to prefer a server in the local site but fail over to the remote site: + +```text +# Site A has two servers - server-1 is beefier, give it 75% of the load: +_x-puppet._tcp.site-a.example.com. IN SRV 0 75 8140 server-1.site-a.example.com. +_x-puppet._tcp.site-a.example.com. IN SRV 0 25 8140 server-2.site-a.example.com. +_x-puppet._tcp.site-a.example.com. IN SRV 1 5 8140 server.site-b.example.com. + +# For site B, prefer the local server unless it's down, then fail back to site A +_x-puppet._tcp.site-b.example.com. IN SRV 0 5 8140 server.site-b.example.com. +_x-puppet._tcp.site-b.example.com. IN SRV 1 75 8140 server-1.site-a.example.com. +_x-puppet._tcp.site-b.example.com. IN SRV 1 25 8140 server-2.site-a.example.com. +``` + +## Centralizing the Certificate Authority + +Additional OpenVox Servers should only share the burden of compiling and serving catalogs, which is why they're +typically referred to as "compilers". Any certificate authority functions should be delegated to a single server. + +Before you centralize this functionality, ensure that the single server that you want to use as the central CA +is reachable at a unique hostname other than (or in addition to) `puppet`. Next, point all agent requests to +the centralized CA server, either by configuring each agent or through DNS `SRV` records. + +### Directing individual agents to a central CA + +On every agent, set the `ca_server` setting in `puppet.conf` (in the `[main]` configuration block) to the +hostname of the server acting as the certificate authority. If you have a large number of existing nodes, it is +easiest to do this by managing `puppet.conf` with an OpenVox module and a template. + +> **Note:** Set this setting _before_ provisioning new nodes, or they won't be able to complete their initial agent run. + +### Pointing DNS `SRV` records at a central CA + +If you [use `SRV` records for agents](#using-dns-srv-records), you can use the `_x-puppet-ca._tcp.$srv_domain` +DNS name to point clients to one specific CA server, while the `_x-puppet._tcp.$srv_domain` DNS name handles +most of their requests and can point to a set of compilers. + +## Creating and configuring compilers + +To add a compiler to your deployment, begin by [installing and configuring OpenVox Server](./install_from_packages.html) on it. + +Before running `puppet agent` or starting `puppetserver` on the new compiler: + +1. In the compiler's `puppet.conf`, in the `[main]` configuration block, set `ca_server` to the hostname of + the server acting as the certificate authority. + +1. In the compiler's `webserver.conf` file, add and set the following SSL settings: + - ssl-cert + - ssl-key + - ssl-ca-cert + - ssl-crl-path + +1. [Disable OpenVox Server's certificate authority services](./configuration.html#service-bootstrapping). + If an `ssldir` is configured, make sure it's set in the `[main]` block of `puppet.conf` only. + +1. If you're using the [DNS round robin method](#using-round-robin-dns) of agent load balancing, or a + [load balancer](#using-a-hardware-load-balancer) in TCP proxying mode, provide compilers with certificates + using DNS Subject Alternative Names. + + Configure `dns_alt_names` in the `[main]` block of `puppet.conf` to cover every DNS name that might be + used by an agent to access this server. + + ```text + dns_alt_names = puppet,puppet.example.com,puppet.site-a.example.com + ``` + + If the agent or server has been run and already created a certificate, remove it by running + `sudo puppet ssl clean`. If an agent has requested a certificate from the server, delete it there to + re-issue a new one with the alt names: `puppetserver ca clean server-2.example.com`. + +1. Request a new certificate by running `puppet agent --test --waitforcert 10`. + +1. Log into the CA server and run `puppetserver ca sign server-2.example.com`. + +## Centralizing reports and exported resources + +If you use an HTTP report processor, point your primary server and all compilers at the same shared report +server in order to see all of your agents' reports. + +If you use exported resources, use OpenVoxDB and point your primary server and all compilers at a shared +OpenVoxDB instance. A reasonably robust OpenVoxDB server can handle many compilers and many thousands of agents. + +See the [OpenVoxDB documentation](/openvoxdb/latest/) for instructions on deploying an OpenVoxDB +server, then configure every compiler to use it. Note that every server and compiler must have its own +[certificate allowlist entry](/openvoxdb/latest/configure.html) if you're using HTTPS certificates +for authorization. + +## Keeping manifests and modules synchronized across compilers + +You must ensure that all compilers have identical copies of your manifests, modules, and external node +classifier data. + +The recommended approach is to use [r10k](https://github.com/voxpupuli/r10k) with a webhook to trigger +automatic code deployment to all compilers on every push to your control repository. The Vox Pupuli +[webhook-go](https://github.com/voxpupuli/webhook-go) service provides a lightweight webhook receiver +that calls r10k on each compiler. See the [load balanced architecture guide](https://voxpupuli.org/docs/arch_load_balanced) +for a full reference setup. + +Other options include: + +- Running r10k manually or via `cron` on each compiler. +- Running an out-of-band `rsync` task via `cron`. +- Configuring `puppet agent` on each compiler to point to a designated primary server, then use OpenVox itself + to distribute the modules. diff --git a/docs/_openvox-server_9x/services_puppetserver.markdown b/docs/_openvox-server_9x/services_puppetserver.markdown new file mode 100644 index 000000000..ff11799eb --- /dev/null +++ b/docs/_openvox-server_9x/services_puppetserver.markdown @@ -0,0 +1,167 @@ +--- +layout: default +title: "OpenVox Server: Services overview" +--- + +[external_ca]: ./intermediate_ca.html + +OpenVox is configured in an agent-server architecture, in which a primary server manages configuration for a fleet of managed agent nodes. +OpenVox Server performs the role of the primary server: it compiles catalogs, serves files, manages certificates, and receives reports from agents. +It is a Ruby and Clojure application that runs on the Java Virtual Machine (JVM), executing catalog compilation inside JRuby interpreters +with supporting services written in Clojure, coordinated by the Trapperkeeper service framework. + +This page describes the run environment and architecture of OpenVox Server. For practical instructions, see the docs for [installing](./install_from_packages.html) and [configuring](./configuration.html) it. + +## Supported Platforms + +OpenVox Server packages are available for Debian, Ubuntu, Red Hat Enterprise Linux and its derivatives, Amazon Linux, Fedora, SLES, and RHEL FIPS. +For the full list of supported distributions, versions, and architectures, see [Before you begin](./install_pre.html). + +OpenVox Server requires Java 21 or 25. It does not bundle a JDK; install one from your distribution's repositories before installing OpenVox Server. + +OpenVox Server releases are versioned separately from OpenVox (the agent). Major versions are aligned: OpenVox Server 9.x is compatible with OpenVox 9.x, and the +`openvox-server` 9.x package depends on `openvox-agent` 9.x on the same host. + +## Controlling the Service + +The OpenVox Server service name is `puppetserver`. On systemd-based platforms: + +```shell +systemctl start puppetserver +systemctl stop puppetserver +systemctl restart puppetserver +systemctl status puppetserver +systemctl enable puppetserver # start automatically on boot +``` + +On older SysV-style init systems, use `service puppetserver start|stop|restart|status` instead. + +## OpenVox Server's Run Environment + +OpenVox Server consists of several related services that share state and route requests among themselves. These services run inside a single JVM process using the Trapperkeeper service framework. +The main visible consequence of this architecture is the set of separate config files that manage the different internal services. + +### Embedded Web Server + +OpenVox Server uses a Jetty-based web server embedded in the service's JVM process. No additional configuration is needed for basic operation; it works out of the box and performs well under production-level loads. + +The web server's settings can be modified in [`webserver.conf`](./config_file_webserver.html). You may need to edit this file if you're [using an external CA][external_ca] or running on a non-standard port. + +### OpenVox API Service + +OpenVox Server includes a service that handles agent configuration requests. See [OpenVox HTTP API](./http_api_index.html) for documentation on the core APIs. + +For OpenVox Server-specific APIs hosted by this service, see: + +- [The `environment_classes` endpoint](./puppet-api/v3/environment_classes.html) +- [The `environment_modules` endpoint](./puppet-api/v3/environment_modules.html) + +### Certificate Authority Service + +OpenVox Server includes a certificate authority (CA) service that: + +- Accepts certificate signing requests (CSRs) from nodes +- Serves certificates and a certificate revocation list (CRL) to nodes +- Optionally accepts commands to sign or revoke certificates (disabled by default) + +The relevant endpoints are `certificate`, `certificate_request`, `certificate_revocation_list`, and `certificate_status`. See [CA HTTP API](./http_api_index.html#ca-v1-http-api) for details. + +Signing and revoking certificates over the network is disallowed by default. You can use [`auth.conf`](./config_file_auth.html) to allow specific certificate owners to issue commands. + +The CA service stores credentials as `.pem` files under `/etc/puppetlabs/puppetserver/ca/`. Use the `puppetserver ca` command to list, sign, and revoke certificates. +See [`ca.conf`](./config_file_ca.html) for CA configuration options. + +### Admin API Service + +OpenVox Server includes an administrative API for triggering maintenance tasks. + +The primary use case is deploying new Puppet code without restarting the service: the environment cache endpoint expires all cached environments, +causing OpenVox Server to pick up code changes on the next agent run. + +For API docs, see: + +- [The `environment-cache` endpoint](./admin-api/v1/environment-cache.html) +- [The `jruby-pool` endpoint](./admin-api/v1/jruby-pool.html) + +For details about environment caching, see [Environments](/openvox/latest/environments_about.html). + +### JRuby Interpreters + +Most of OpenVox Server's work — compiling catalogs, receiving reports, etc. — is done by Ruby code. Rather than using the OS's MRI Ruby runtime, +OpenVox Server runs this code in JRuby, an implementation of the Ruby interpreter that runs on the JVM. + +Because OpenVox Server does not use system Ruby, you cannot use the system `gem` command to install Ruby gems for use by OpenVox modules or extensions. +Instead, use the `puppetserver gem` command. See [Using Ruby Gems](./gems.html) for details. + +The `puppetserver ruby` and `puppetserver irb` commands run Ruby code in a JRuby environment and are useful for testing or debugging code that will run on the server. +The `JAVA_ARGS_CLI` environment variable controls Java arguments passed to these commands +(set it in `/etc/sysconfig/puppetserver` or `/etc/default/puppetserver`). See [Subcommands](./subcommands.html) for details. + +To handle parallel requests, OpenVox Server maintains several JRuby interpreters, each independently running OpenVox's application code, and distributes agent requests among them. + +Configure the JRuby interpreters in the `jruby-puppet` section of [`puppetserver.conf`](./config_file_puppetserver.html). + +#### Tuning Guide + +You can maximize OpenVox Server's performance by tuning your JRuby configuration. See the [Tuning Guide](./tuning_guide.html) for details. + +### User + +OpenVox Server runs as the `puppet` user. This is specified in `/etc/sysconfig/puppetserver` on RPM-based systems, +or `/etc/default/puppetserver` on Debian-based systems. OpenVox Server ignores the `user` and `group` settings in `puppet.conf`. + +All of OpenVox Server's files and directories must be readable and writable by the `puppet` user. + +### Ports + +By default, OpenVox Server listens on TCP port **8140** for HTTPS traffic. Your OS and firewall must allow the JVM process to accept incoming connections on this port. + +You can change the port in `webserver.conf` if necessary. See [webserver.conf](./config_file_webserver.html) for details. + +### Logging + +All of OpenVox Server's logging is routed through the JVM [Logback](http://logback.qos.ch/) library. Log files are written to `/var/log/puppetlabs/puppetserver/`: + +- `puppetserver.log` — main log, default level INFO +- `puppetserver-access.log` — HTTP access log +- `puppetserver-status.log` — status API requests + +By default, OpenVox Server sends nothing to syslog. All log messages follow the same path, including HTTP traffic, catalog compilation, and certificate processing. + +Logback archives log files when they exceed 200 MB, and automatically deletes the oldest logs when the total size of all server logs exceeds 1 GB. + +Logback is highly configurable via [`logback.xml`](./config_file_logbackxml.html); see [the configuration docs](./configuration.html#logging) for details on customizing log output. + +Errors that occur before logging is set up, or that cause the logging system to fail, appear in `journalctl` on systemd-based platforms. + +### SSL Termination + +By default, OpenVox Server handles SSL termination automatically. + +In network configurations that require external SSL termination (e.g. with a hardware load balancer), additional configuration is needed. +See the [External SSL Termination](./external_ssl_termination.html) page for details. In summary: + +- Configure OpenVox Server to use HTTP instead of HTTPS. +- Configure OpenVox Server to accept SSL information via HTTP headers. +- Secure your network so that OpenVox Server **cannot** be directly reached by **any** untrusted clients. +- Configure your SSL-terminating proxy to set these HTTP headers: + - `X-Client-Verify` (mandatory) + - `X-Client-DN` (mandatory for client-verified requests) + - `X-Client-Cert` (optional; required for [trusted facts](/openvox/latest/lang_facts_and_builtin_vars.html)) + +## Configuring OpenVox Server + +OpenVox Server uses a combination of OpenVox's standard config files and its own configuration files located in the `conf.d` directory. + +The `conf.d` directory contains: + +- `global.conf` — global configuration settings +- `webserver.conf` and `web-routes.conf` — web server settings +- `puppetserver.conf` — JRuby interpreter and admin API settings +- `auth.conf` — authentication rules for server endpoints +- `ca.conf` — certificate authority settings + +For full details, see the [Configuration](./configuration.html) page. + +OpenVox Server also uses OpenVox's standard config files, including most settings in [`puppet.conf`](/openvox/latest/config_file_main.html). +However, some `puppet.conf` settings are treated differently by OpenVox Server — see [puppet.conf differences](./puppet_conf_setting_diffs.html) for details. diff --git a/docs/_openvox-server_9x/ssl_server_certificate_change_and_virtual_ips.markdown b/docs/_openvox-server_9x/ssl_server_certificate_change_and_virtual_ips.markdown new file mode 100644 index 000000000..addc5d082 --- /dev/null +++ b/docs/_openvox-server_9x/ssl_server_certificate_change_and_virtual_ips.markdown @@ -0,0 +1,44 @@ +--- +layout: default +title: "Puppet Server: Known Issues: SSL Server Certificate Change and Virtual IP Addresses" +canonical: "/puppetserver/latest/ssl_server_certificate_change_and_virtual_ips.html" +--- + +Puppet Server can often encounter `server certificate change is restricted` errors when it makes HTTPS requests to a group of load-balanced servers behind a virtual IP address. This page describes the issue, workarounds for the issue, and our future plans for handling the issue. + +The behavior described in this page was identified in [SERVER-207](https://tickets.puppet.com/browse/SERVER-207). + +## Summary of the Problem + +The JDK handles HTTPS client connections differently from Ruby, so Puppet Server has some behaviors that you wouldn't see with a Passenger-based Puppet master. + +Specifically, if Puppet Server makes multiple HTTPS requests to the same server, it attempts to resume an SSL session using the session ID provided from the server. If that server doesn't have a suitable session ID, Puppet Server and the server try to renegotiate the session. + +During the renegotiation, Puppet Server checks to make sure the server is using the same certificate (to [mitigate the TLS triple handshake attack](https://secure-resumption.com)). If that check fails, it aborts the connection. + +For example, if Puppet Server is configured to use a load-balanced group of PuppetDB servers, and those servers all use different certificates, some of the certificate checks will fail, and Puppet Server will abort those connections. + +These connection failures may look like this in the `puppetserver.log` file: + +~~~text +2014-11-20 22:04:03,392 ERROR [c.p.h.c.SyncHttpClient] Error executing http request +javax.net.ssl.SSLHandshakeException: server certificate change is restricted during renegotiation +~~~ + +## Working Around the Problem + +### Recommended Workaround + +If you need Puppet Server to act as a client to a load-balanced HTTPS service (e.g., multiple PuppetDB servers), your best option right now is to have all of the servers behind the load balancer present the same certificate. + +There appear to be ways to fulfill the renegotiation check with certificates that only partially match ([see here for more info](http://hg.openjdk.java.net/bsd-port/bsd-port/jdk/rev/eabde5c42157#l1.186)), but these might not be foolproof, especially because future JDK implementations might disallow these partial matches. The most reliable way is to simply use the same certificates. + +### Alternate Workaround + +It's also possible to configure the JDK to allow server certificate changes. You can do this by editing the `/etc/sysconfig/puppetserver` file and adding `-Djdk.tls.allowUnsafeServerCertChange=true` to the value of the `JAVA_ARGS` variable. + +We don't recommend this workaround, however, because it can make Puppet Server more vulnerable to the TLS triple handshake attack. + +The use of the `allowUnsafeServerCertChange` property is documented in +. + diff --git a/docs/_openvox-server_9x/status-api/v1/services.markdown b/docs/_openvox-server_9x/status-api/v1/services.markdown new file mode 100644 index 000000000..3d38e36f7 --- /dev/null +++ b/docs/_openvox-server_9x/status-api/v1/services.markdown @@ -0,0 +1,102 @@ +--- +layout: default +title: "OpenVox Server: Status API: Services" +--- + +[`auth.conf`]: ../../config_file_auth.html + +The `services` endpoint of OpenVox Server's Status API provides information about services running on OpenVox Server. The endpoint provides information about memory usage similar to the +data produced by the Java MemoryMXBean, as well as basic data on the `pupppetserver` process's state and uptime. See the +[Java MemoryMXBean documentation](https://docs.oracle.com/javase/7/docs/api/java/lang/management/MemoryMXBean.html) for help interpreting the memory information. + +> **Note:** This is an experimental feature provided for troubleshooting purposes. In future releases, the `services` endpoint's response payload might change without warning. +> +> For information about HTTP client metrics, which are served from the status endpoint, see [their documentation](../../http_client_metrics.html). + +## `GET /status/v1/services` + +### Supported HTTP methods + +GET + +### Supported formats + +JSON + +### Query parameters + +- `level` (optional): The response includes status information for all registered services at the requested level of detail. Default: `info`. Valid values: + - `critical`: Returns the minimum amount of status information for each service. This level returns data quickly and is suitable for frequently updating uses, such as health checks for a load balancer. + + - `info`: Returns more info than the `critical` level for each service. The specific data depends on the implementation details of the services loaded in the application, but generally includes enough + human-readable data to provide a quick impression of each service's health and status. + + - `debug`: This level returns status information about a service in enough detail to be suitable for debugging issues with the `puppetserver` process. Depending on the service, this level can be significantly + more expensive than lower levels, reduce the process's performance, and generate large amounts of data. This level is suitable for producing aggregate metrics about the performance or resource usage of + OpenVox Server's subsystems. + + The information returned for any service at each increasing level of detail includes the data from lower levels. In other words, the `info` level returns the same data structure as the `critical` level, and + might provide additional data in the `status` field depending on the service. Likewise, the `debug` level returns the same data structure as `info`, and might also add additional information in the `status` + field. + +### Response + +The `services` endpoint's response includes information for each service about which the Status service is aware. Each service's `state` value is one of the following: + +- `running`, if and only if all services are running +- `error` if any service reports an error +- `starting` if any service reports that it is starting, and no service reports an error or that it is stopping +- `stopping` if any service reports that it is stopping and no service reports an error +- `unknown` if any service reports an unknown state and no services report an error + +Requests to this endpoint return one of the following status codes: + +- 200 when all services are in `running` state. +- 404 when a requested service is not found. +- 503 when the service state is `unknown`, `error`, `starting`, or `stopping` + +### Example request and response for a debug-level GET request + +```text +GET /status/v1/services?level=debug + +HTTP/1.1 200 OK +Content-Type: application/json + +{ + "status-service": { + "detail_level": "debug", + "service_status_version": 1, + "service_version": "0.3.5", + "state": "running", + "status": { + "experimental": { + "jvm-metrics": { + "heap-memory": { + "committed": 1049100288, + "init": 268435456, + "max": 1908932608, + "used": 216512656 + }, + "non-heap-memory": { + "committed": 256466944, + "init": 2555904, + "max": -1, + "used": 173201432 + }, + "start-time-ms": 1472496731281, + "up-time-ms": 538974 + } + } + } + } +} +``` + +### Authorization + +Requests to the `services` endpoint are authorized by the [Trapperkeeper-based authorization process][`auth.conf`]. For more information about the supported OpenVox Server authorization +processes and configuration settings, see the [`auth.conf` documentation][`auth.conf`]. + +One may also restrict access to the status service by changing the `client-auth` setting to `required` for the webserver. See +[Configuring the Webserver Service](https://github.com/openvoxproject/trapperkeeper-webserver/blob/main/doc/jetty-config.md#client-auth) for more information on the `client-auth` setting. diff --git a/docs/_openvox-server_9x/status-api/v1/simple.markdown b/docs/_openvox-server_9x/status-api/v1/simple.markdown new file mode 100644 index 000000000..0978ebdd5 --- /dev/null +++ b/docs/_openvox-server_9x/status-api/v1/simple.markdown @@ -0,0 +1,63 @@ +--- +layout: default +title: "OpenVox Server: Status API: Simple" +--- + +[`auth.conf`]: ../../config_file_auth.html + +The `simple` endpoint of OpenVox Server's Status API provides a simple +indication of whether OpenVox Server is running on a server. It's designed for +load balancers that don't support any kind of JSON parsing or parameter setting +and returns a simple string body (either the state of the server or a simple +error message) and a status code relevant to the result. + +The content type for this endpoint is `text/plain; charset=utf-8`. + +## `GET /status/v1/simple` + +### Supported HTTP methods + +GET + +### Supported formats + +Plain text + +### Query parameters + +None + +### Response + +The `simple` endpoint's response consists of a single word describing OpenVox Server's status: + +- `running`, if and only if the OpenVox Server service is running +- `error`, if the service reports an error +- `unknown`, if the service reports an unknown state, but doesn't report an error + +Requests to this endpoint return one of the following status codes: + +- 200 if and only if the OpenVox Server service reports a status of running +- 503 if the service's status is unknown or error + +### Example request and response for a GET request + +```text +GET /status/v1/simple + +HTTP/1.1 200 OK +Content-Type: application/json + +running +``` + +### Authorization + +Requests to the `simple` endpoint are authorized by the +[Trapperkeeper-based authorization process][`auth.conf`]. For more information about the supported OpenVox Server authorization processes and configuration settings, see the +[`auth.conf` documentation][`auth.conf`]. + +One may also restrict access to the status service by changing the +`client-auth` setting to `required` for the webserver. See +[Configuring the Webserver Service](https://github.com/openvoxproject/trapperkeeper-webserver/blob/main/doc/jetty-config.md#client-auth) +for more information on the `client-auth` setting. diff --git a/docs/_openvox-server_9x/subcommands.markdown b/docs/_openvox-server_9x/subcommands.markdown new file mode 100644 index 000000000..83637f9a1 --- /dev/null +++ b/docs/_openvox-server_9x/subcommands.markdown @@ -0,0 +1,214 @@ +--- +layout: default +title: "Puppet Server: Subcommands" +canonical: "/puppetserver/latest/subcommands.html" +--- + +We've provided several CLI commands to help with debugging and exploring Puppet Server. Most of the commands are the same ones you would use in a Ruby environment --- such as `gem`, `ruby`, and `irb` --- except +they run against Puppet Server's JRuby installation and gems instead of your system Ruby. + +The following subcommands are provided: + +- [ca](#ca) +- [gem](#gem) +- [ruby](#ruby) +- [irb](#irb) +- [foreground](#foreground) + +The format for each subcommand is: + +```sh +puppetserver [] +``` + +When running from source, the format is: + +```sh +lein -c /path/to/puppetserver.conf [--] [] +``` + +Note that if you are running from source, you need to separate flag arguments (such as `--version` or `-e`) with `--`, as shown above. Otherwise, those arguments will be applied to Leiningen instead of to +Puppet Server. This isn't necessary when running from packages (i.e., `puppetserver `). + +## ca + +## Available actions + +CA subcommand usage: `puppetserver ca [options]` The available actions: + +- `clean`: clean files from the CA for certificates +- `generate`: create a new certificate signed by the CA +- `setup`: generate a root and intermediate signing CA for Puppet Server +- `import`: import the CA's key, certs, and CRLs +- `list`: list all certificate requests +- `revoke`: revoke a given certificate +- `sign`: sign a given certificate + +Because these commands utilize Puppet Server’s API, all except `setup` and `import` need the server to be running in order to work. + +Because these commands are shipped as a gem alongside Puppet Server, it can be updated out-of-band to pick up improvements and bug fixes. To upgrade it, run this command: +`/opt/puppetlabs/puppet/bin/gem install -i /opt/puppetlabs/puppet/lib/ruby/vendor_gems puppetserver-ca` + +**Note:** These commands are available in Puppet 5, but in order to use them, you must update Puppet Server’s `auth.conf` to include a rule allowing the master’s certname to access the `certificate_status` and +`certificate_statuses` endpoints. The same applies to upgrading in open source Puppet: if you're upgrading from Puppet 5 to Puppet 6 and are not regenerating your CA, you must whitelist the master’s certname. +See [Puppet Server Configuration Files: auth.conf](./config_file_auth.html) for details on how to use `auth.conf`. + +Example: + +```text +{ + # Allow the CA CLI to access the certificate_status endpoint + match-request: { + path: "/puppet-ca/v1/certificate_status" + type: path + method: [get, put, delete] + } + allow: master.example.com + sort-order: 500 + name: "puppetlabs cert status" +}, +``` + +### Signing certs with SANs or auth extensions + +With the removal of `puppet cert sign`, it's possible for Puppet Server’s CA API to sign certificates with subject alternative names or auth extensions, which was previously completely disallowed. This is +disabled by default for security reasons, but you can turn it on by setting `allow-subject-alt-names` or `allow-authorization-extensions` to true in the `certificate-authority` section of Puppet Server’s config +(usually located in `ca.conf`). After these have been configured, you can use `puppetserver ca sign --certname ` to sign certificates with these additions. + +## gem + +Installs and manages gems that are isolated from system Ruby and are accessible only to Puppet Server. This is a simple wrapper around the standard Ruby `gem`, so all of the usual arguments and flags should +work as expected. + +Examples: + +```sh +puppetserver gem install pry --no-ri --no-rdoc +``` + +```sh +lein gem -c /path/to/puppetserver.conf -- install pry --no-ri --no-rdoc +``` + +If needed, you also can use the `JAVA_ARGS_CLI` environment variable to pass along custom arguments to the Java process that the `gem` command is run within. + +Example: + +```sh +JAVA_ARGS_CLI=-Xmx8g puppetserver gem install pry --no-ri --no-rdoc +``` + +If you prefer to have the `JAVA_ARGS_CLI` option persist for multiple command executions, you could set the value in the `/etc/sysconfig/puppetserver` or `/etc/default/puppetserver` file, depending upon your OS +distribution: + +```ini +JAVA_ARGS_CLI=-Xmx8g +``` + +With the value specified in the sysconfig or defaults file, subsequent commands would use the `JAVA_ARGS_CLI` variable automatically: + +```sh +$ puppetserver gem install pry --no-ri --no-rdoc +// Would run 'gem' with a maximum Java heap of 8g +``` + +For more information, see [Puppet Server and Gems](./gems.html). + +## ruby + +Runs code in Puppet Server's JRuby interpreter. This is a simple wrapper around the standard Ruby `ruby`, so all of the usual arguments and flags should work as expected. + +Useful when experimenting with gems installed via `puppetserver gem` and the Puppet and Puppet Server Ruby source code. + +Examples: + +```sh +puppetserver ruby -e "require 'puppet'; puts Puppet[:certname]" +``` + +```sh +lein ruby -c /path/to/puppetserver.conf -- -e "require 'puppet'; puts Puppet[:certname]" +``` + +If needed, you also can use the `JAVA_ARGS_CLI` environment variable to pass along custom arguments to the Java process that the `ruby` command is run within. + +Example: + +```sh +JAVA_ARGS_CLI=-Xmx8g puppetserver ruby -e "require 'puppet'; puts Puppet[:certname]" +``` + +If you prefer to have the `JAVA_ARGS_CLI` option persist for multiple command executions, you could set the value in the `/etc/sysconfig/puppetserver` or `/etc/default/puppetserver` file, depending upon your OS +distribution: + +```ini +JAVA_ARGS_CLI=-Xmx8g +``` + +With the value specified in the sysconfig or defaults file, subsequent commands would use the `JAVA_ARGS_CLI` variable automatically: + +```sh +$ puppetserver ruby -e "require 'puppet'; puts Puppet[:certname]" +// Would run 'ruby' with a maximum Java heap of 8g +``` + +## irb + +Starts an interactive REPL for the JRuby that Puppet Server uses. This is a simple wrapper around the standard Ruby `irb`, so all of the usual arguments and flags should work as expected. + +Like the `ruby` subcommand, this is useful for experimenting in an interactive environment with any installed gems (via `puppetserver gem`) and the Puppet and Puppet Server Ruby source code. + +Examples: + +```ruby +$ puppetserver irb +irb(main):001:0> require 'puppet' +=> true +irb(main):002:0> puts Puppet[:certname] +centos6-64.localdomain +=> nil +``` + +```sh +$ lein irb -c /path/to/puppetserver.conf -- --version +irb 0.9.6(09/06/30) +``` + +If needed, you also can use the `JAVA_ARGS_CLI` environment variable to pass along custom arguments to the Java process that the `irb` command is run within. + +Example: + +```sh +JAVA_ARGS_CLI=-Xmx8g puppetserver irb +``` + +If you prefer to have the `JAVA_ARGS_CLI` option persist for multiple command executions, you could set the value in the `/etc/sysconfig/puppetserver` or `/etc/default/puppetserver` file, depending upon your OS +distribution: + +```ini +JAVA_ARGS_CLI=-Xmx8g +``` + +With the value specified in the sysconfig or defaults file, subsequent commands would use the `JAVA_ARGS_CLI` variable automatically: + +```sh +$ puppetserver irb +// Would run 'irb' with a maximum Java heap of 8g +``` + +## foreground + +Starts the Puppet Server, but doesn't background it; similar to starting the service and then tailing the log. + +Accepts an optional `--debug` argument to raise the logging level to DEBUG. + +Examples: + +```java +$ puppetserver foreground --debug +2014-10-25 18:04:22,158 DEBUG [main] [p.t.logging] Debug logging enabled +2014-10-25 18:04:22,160 DEBUG [main] [p.t.bootstrap] Loading bootstrap config from specified path: '/etc/puppetserver/bootstrap.cfg' +2014-10-25 18:04:26,097 INFO [main] [p.s.j.jruby-puppet-service] Initializing the JRuby service +2014-10-25 18:04:26,101 INFO [main] [p.t.s.w.jetty-service] Initializing web server(s). +2014-10-25 18:04:26,149 DEBUG [clojure-agent-send-pool-0] [p.s.j.jruby-puppet-agents] Initializing JRubyPuppet instances with the following settings: +``` diff --git a/docs/_openvox-server_9x/tuning_guide.markdown b/docs/_openvox-server_9x/tuning_guide.markdown new file mode 100644 index 000000000..1832183e9 --- /dev/null +++ b/docs/_openvox-server_9x/tuning_guide.markdown @@ -0,0 +1,169 @@ +--- +layout: default +title: "Puppet Server: Tuning Guide" +canonical: "/puppetserver/latest/tuning_guide.html" +--- + +Puppet Server provides many configuration options that can be used to tune the +server for maximum performance and hardware resource utilization. In this guide, +we'll highlight some of the most important settings that you can use to get +the best performance in your environment. + +## Puppet Server and JRuby + +Before you begin tuning your configuration, it's helpful to have a little bit +of context on how Puppet Server uses JRuby to handle incoming HTTP requests from +your Puppet agents. + +When Puppet Server starts up, it creates a pool of JRuby interpreters to use +as workers when it needs need to execute some of the Puppet Ruby code. You can think +of these almost as individual Ruby "virtual machines" that are controlled by +Puppet Server; it's not entirely dissimilar to the way that Passenger spawns +several Ruby processes to hand off work to. + +Puppet Server isolates these JRuby instances so that they will only be allowed +to handle one request at a time. This ensures that we don't encounter any +concurrency issues, because the Ruby code is not thread-safe. When an HTTP request +comes in to Puppet Server, and it determines that some Ruby code will need to be +executed in order to handle the request, Puppet Server "borrows" a JRuby instance +from the pool, uses it to do the work, and then "returns" it to the pool. If +there are no JRuby instances available in the pool at the time a request +comes in (presumably because all of the JRuby instances are already in use handling +other requests), Puppet Server will block the request until one becomes available. + +(In the future, this approach will allow us to do some really powerful things +such as creating multiple pools of JRubies and isolating each of your Puppet +environments to a single pool, to ensure that there is no pollution from one +Puppet environment to the next.) + +This brings us to the two most important settings that you can use to tune your +Puppet Server. + +### Number of JRubies + +The most important setting that you can use to improve the throughput of your +Puppet Server installation is the [`max-active-instances`](./config_file_puppetserver.html) +setting. The value of this setting is used by Puppet Server to determine how +many JRuby instances to create when the server starts up. + +From a practical perspective, this setting basically controls how many Puppet +agent runs Puppet Server can handle concurrently. The minimum value you can +get away with here is `1`, and if your installation is small enough that +you're unlikely to ever have more than one Puppet agent checking in with the +server at exactly the same time, this is totally sufficient. + +However, if you specify a value of `1` for this setting, and then you have two +Puppet agent runs hitting the server at the same time, the requests being made by the +second agent will be effectively blocked until the server has finished handling all of the +requests from the first agent. In other words, one of Puppet Server's threads will have +"borrowed" the single JRuby instance from the pool to handle the requests from the first +agent, and only when those requests are completed will it return the JRuby instance +to the pool. At that point, the next thread can "borrow" the JRuby instance to +use to handle the requests from the second agent. + +Assuming you have more than one CPU core in your machine, this situation means +that you won't be getting the maximum possible throughput from your Puppet Server +installation. Increasing the value from `1` to `2` would mean that Puppet Server +could now use a second CPU core to handle the requests from a second Puppet agent +simultaneously. + +It follows, then, that the maximum sensible value to use for this setting will +be roughly the number of CPU cores you have in your server. Setting the value +to something much higher than that won't improve performance, because even if there +are extra JRuby instances available in the pool to do work, they won't be able +to actually do any work if all of the CPU cores are already busy using JRuby +instances to handle incoming agent requests. + +(There are some exceptions to this rule. For example, if you have report processors that make a network connection as part of the processing of a report, and if there is a chance +that the network operation is slow and will block on I/O for some period of time, +then it might make sense to have more JRuby instances than the number of cores. The JVM is smart enough to suspend the thread that is handling those kinds of requests and use the CPUs for other work, assuming there are still JRuby instances available in the pool. In a case like this you might want to set `max-active-instances` to a value higher than the number of CPUs.) + +At this point you may be wondering, "What's the downside to just setting +`max-active-instances` to a really high value?" The answer to this question, in +a nutshell, is "memory usage". This brings us to the other extremely important setting to consider for Puppet Server. + +### JVM Heap Size + +The JVM's "max heap size" controls the maximum amount of (heap*[[1]](#footnotes) +memory that the JVM process is allowed to request from the operating system. You +can set this value via the `-Xmx` command-line argument at JVM startup. (In the +case of Puppet Server, you'll find this setting in the "defaults" file for Puppet +Server for your operating system; this will generally be something like +`/etc/sysconfig/puppetserver` or `/etc/defaults/puppetserver`.) + +> **Upgrade note:** If you modified the defaults file in Puppet Server 2.4.x or earlier, +> then lost those modifications or see `Service ':PoolManagerService' not found` warnings +> after upgrading to Puppet Server 2.5, be aware that the package might have attempted to +> overwrite the file during the upgrade. See the [Puppet Server 2.5 release +> notes](https://docs.puppet.com/puppetserver/2.5/release_notes.html) for details. + +If your application's memory usage approaches this value, the JVM will try to +get more aggressive with garbage collection to free up memory. In certain +situations, you may see increased CPU activity related to this garbage collection. If the JVM is unable to recover enough memory to keep the application running +smoothly, you will eventually encounter an `OutOfMemoryError`, and the process +will shut down. + +For Puppet Server, we also use a JVM argument, +`-XX:HeapDumpOnOutOfMemoryError`, to cause the JVM to dump an `.hprof` file to +disk. This is basically a memory snapshot at the point in time where the +error occurred; it can be loaded into various profiling tools to get a better +understanding of where the memory was being used. + +(Note that there is another setting, "min heap size", that is controlled via +the -Xms setting; [Oracle recommends](http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.%20Total%20Heap|outline) setting this value to the same value that you use for -Xmx.) + +The most important factor when determining the max heap size for Puppet Server +is the value of `max-active-instances`. Each JRuby instance needs to load up +a copy of the Puppet Ruby code, and then needs some amount of memory overhead +for all of the garbage that gets generated during a Puppet catalog compilation. +Also, the memory requirements will vary based on how many Puppet modules you +have in your module path, how much Hiera data you have, etc. At this time we +estimate that a reasonable ballpark figure is about 512MB of RAM per JRuby +instance, but that can vary depending on some characteristics of your Puppet +codebase. For example, if you have a really high number of modules or a great +deal of Hiera data, you might find that you need more than 512MB per JRuby +instance. + +You'll also want to allocate a little extra heap to be used by the rest of the +things going on in Puppet Server: the web server, etc. So, a good rule of thumb +might be 512MB + (max-active-instances * 512MB). + +We're working on some optimizations for really small installations (for testing, +demos, etc.). Puppet Server should run fine with a value of 1 for +`max-active-instances` and a heap size of 512MB, and we might be able to improve +that further in the future. + +### Tying Together `max-active-instances` and Heap Size + +We're still gathering data on what the best default settings are, to try to provide +an out-of-the-box configuration that works well in most environments. In versions +prior to 1.0.8 in the 1.x series (compatible with Puppet 3.x), and prior to 2.1.0 +in the 2.x series (compatible with Puppet 4.x), the default +value is `num-cpus + 2`. This value will be far too high if you're running on +a system with a large number of CPU cores. + +As of Puppet Server 1.0.8 and 2.1.0, if you don't provide an explicit value for this setting, +we'll default to `num-cpus - 1`, with a minimum value of `1` and a maximum value of +`4`. The maximum value of `4` is probably too low for production environments +with beefy hardware and a high number of Puppet agents checking in, but our +current thinking is that it's better to ship with a default setting that is too +low and allow you to tune up, than to ship with a default setting that is too +high and causes you to run into `OutOfMemory` errors. In general, +it's recommended that you explicitly set this value to something that you think +is reasonable in your environment. To encourage this, we log a warning +message at startup if you haven't provided an explicit value. + +### Potential JAVA ARGS settings + +If you’re working outside of lab environment, increase `ReservedCodeCache` to `512m` under normal load. If you’re working with 6-12 JRuby instances (or a `max-requests-per-instance` value significantly less than 100k), run with a `ReservedCodeCache` of 1G. Twelve or more JRuby instances in a single server might require 2G or more. + +Similar caveats regarding scaling `ReservedCodeCache` might apply if users are managing `MaxMetaspace`. + +## Footnotes + +[1] The vast majority of the memory footprint of a JVM process can usually be + accounted for by the heap size. However, there is some amount of non-heap + memory that will always be used, and for programs that call out to native + code at all, there may be a bit more. Generally speaking, the resident + memory usage of a JVM process shouldn't exceed the max heap size by more + than 256MB or so, but exceeding the max heap size by some amount is normal. diff --git a/docs/_openvox-server_9x/upgrade_minor.md b/docs/_openvox-server_9x/upgrade_minor.md new file mode 100644 index 000000000..575f4a0e8 --- /dev/null +++ b/docs/_openvox-server_9x/upgrade_minor.md @@ -0,0 +1,65 @@ +--- +layout: default +title: "Upgrading OpenVox Server" +--- + +Use this page for routine OpenVox Server upgrades and for in-place migrations from +legacy Puppet Server packages to OpenVox Server packages. + +OpenVox Server is functionally equivalent to modern Puppet Server. A host cannot have +both Puppet Server and OpenVox Server packages installed at the same time. Back up +`/etc/puppetlabs/` and `/var/opt/puppetlabs/` before you start. + +## Recommended upgrade order + +In an agent-server deployment, upgrade infrastructure components in this order: + +1. `openvox-server` +2. `openvoxdb` +3. `openvoxdb-termini` on server nodes +4. `openvox-agent` on managed nodes + +Keeping the server ahead of the agents it serves avoids compatibility issues during +the upgrade window. + +## Upgrading Linux packages + +On apt-based systems: + +```bash +sudo apt update +sudo apt install --only-upgrade openvox-server +``` + +On yum/dnf-based systems: + +```bash +sudo yum update openvox-server +``` + +After upgrading the server, restart the service to load the new version: + +```bash +sudo systemctl restart puppetserver +sudo systemctl status puppetserver +``` + +## Migrating from legacy Puppet Server packages + +If you are replacing Puppet Server rather than upgrading an existing OpenVox +installation, enable the OpenVox repository first: + +1. Enable the OpenVox repository for your platform — see [OpenVox repositories and packages](/openvox/latest/openvox_platform.html). +2. Install `openvox-server`. The package manager will replace the legacy Puppet Server package. +3. Restore `/etc/puppetlabs/` from your backup if any configuration was lost. +4. Start the service and validate. + +## After the upgrade + +After upgrading: + +1. Confirm the service is running: `sudo systemctl status puppetserver` +2. Check the server log for errors: `sudo journalctl -u puppetserver -n 50` +3. Run a test agent check-in from a managed node: `sudo puppet agent --test` +4. Verify certificate handling and OpenVoxDB connectivity where applicable. +5. Review the [release notes](./release_notes.html) for version-specific changes.