-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Describe the issue
I encountered significant frustration while trying to implement a standardized CI/CD pipeline using Checkov's severity-based filtering. I was attempting to use --hard-fail-on, --soft-fail-on, and related parameters with severity levels (HIGH, CRITICAL, MEDIUM, etc.) to create a graduated failure policy in my pipeline.
The Problem:
After numerous trial-and-error attempts to make severity-based filtering work, the parameters simply had no effect on the scan results. The severity codes were being silently ignored with no indication of why. This led to:
- Wasted time debugging: Hours spent trying different command combinations, syntax variations, and configuration options
- Pipeline confusion: Uncertainty about whether the pipeline was actually enforcing security standards
- Code diving required: Eventually had to dive into Checkov's source code to discover that severity filtering requires a Bridgecrew/Prisma Cloud API key
- Silent failure: No warnings or errors indicated that the parameters were being ignored
This silent failure mode is particularly problematic in CI/CD contexts where you expect explicit failures when security thresholds are exceeded.
Examples
What I was trying to achieve in my standardized pipeline:
# My CI/CD pipeline configuration (unsuccessful attempts)
steps:
- name: Security Scan - High Severity
run: |
checkov -d . --soft-fail --hard-fail-on HIGH,CRITICAL
# Expected: Pipeline fails only if HIGH or CRITICAL issues found
# Actual: Parameters completely ignored, no warning givenCommand-line attempts that failed silently:
# Attempt 1: Using --hard-fail-on with severities
checkov -d . --hard-fail-on HIGH,CRITICAL
# Result: No effect, no warning
# Attempt 2: Using --soft-fail-on with severities
checkov -d . --soft-fail-on MEDIUM,LOW
# Result: No effect, no warning
# Attempt 3: Combining check IDs with severities
checkov -d . --check CKV_AWS_1,HIGH,CRITICAL --hard-fail-on HIGH
# Result: Only CKV_AWS_1 processed, severities silently ignored
# Attempt 4: Different severity syntax variations
checkov -d . --check "HIGH,CRITICAL"
checkov -d . --check HIGH CRITICAL
checkov -d . --check high,critical
# Result: All silently ignored, no indication of the problemWhat SHOULD happen (proposed solution):
$ checkov -d . --hard-fail-on HIGH,CRITICAL
⚠️ WARNING: Severity codes cannot be used without an API key.
Parameters --hard-fail-on contain severity codes: CRITICAL, HIGH
These parameters will be ignored during this scan.
Configure an API key with --bc-api-key to use severity-based filtering.
# Scan continues with check IDs only...Expected behavior with API key (working correctly):
$ checkov -d . --hard-fail-on HIGH,CRITICAL --bc-api-key <key>
# No warning, severity filtering works as expectedVersion:
- Checkov Version: 3.2.495 (but issue exists in earlier versions)
- Python Version: 3.12
- Context: CI/CD pipeline standardization across multiple projects
Additional context
Why this matters:
-
DevOps Experience: When building standardized pipelines, silent failures are the worst kind of bug. You think your security controls are working, but they're not.
-
Documentation gap: The documentation mentions severity codes but doesn't prominently explain the API key requirement for these features to function.
-
Time waste: After spending hours on trial-and-error, I had to read through the source code to understand the dependency on Bridgecrew/Prisma Cloud API for severity information.
-
User confusion: Parameters like
--hard-fail-onand--soft-fail-onappear to be core CLI features, but their severity-based functionality is actually platform-dependent.
What makes this issue important:
- Silent failures are dangerous in security tooling - users assume controls are working when they're not
- Wasted developer time - I'm likely not the only one who's gone through this frustration
- Better UX - A simple warning would have saved hours of debugging
- CI/CD reliability - Pipelines need explicit feedback when configuration is ineffective
Proposed implementation:
I've already implemented this feature in a branch and tested it thoroughly. The solution:
- Detects severity codes in filtering parameters
- Issues clear warnings when API key is missing
- Explains that parameters will be ignored
- Supports all documented formats (including CSV)
- Maintains backward compatibility (warning only, no behavior changes)
- Comprehensive test coverage (13 unit tests)
This would prevent future users from experiencing the same trial-and-error frustration I went through.