Skip to content

feat(general): Add warnings when API-dependent parameters are used without API key #7379

@jasonouellet

Description

@jasonouellet

Describe the issue

I encountered significant frustration while trying to implement a standardized CI/CD pipeline using Checkov's severity-based filtering. I was attempting to use --hard-fail-on, --soft-fail-on, and related parameters with severity levels (HIGH, CRITICAL, MEDIUM, etc.) to create a graduated failure policy in my pipeline.

The Problem:
After numerous trial-and-error attempts to make severity-based filtering work, the parameters simply had no effect on the scan results. The severity codes were being silently ignored with no indication of why. This led to:

  1. Wasted time debugging: Hours spent trying different command combinations, syntax variations, and configuration options
  2. Pipeline confusion: Uncertainty about whether the pipeline was actually enforcing security standards
  3. Code diving required: Eventually had to dive into Checkov's source code to discover that severity filtering requires a Bridgecrew/Prisma Cloud API key
  4. Silent failure: No warnings or errors indicated that the parameters were being ignored

This silent failure mode is particularly problematic in CI/CD contexts where you expect explicit failures when security thresholds are exceeded.

Examples

What I was trying to achieve in my standardized pipeline:

# My CI/CD pipeline configuration (unsuccessful attempts)
steps:
  - name: Security Scan - High Severity
    run: |
      checkov -d . --soft-fail --hard-fail-on HIGH,CRITICAL
      # Expected: Pipeline fails only if HIGH or CRITICAL issues found
      # Actual: Parameters completely ignored, no warning given

Command-line attempts that failed silently:

# Attempt 1: Using --hard-fail-on with severities
checkov -d . --hard-fail-on HIGH,CRITICAL
# Result: No effect, no warning

# Attempt 2: Using --soft-fail-on with severities  
checkov -d . --soft-fail-on MEDIUM,LOW
# Result: No effect, no warning

# Attempt 3: Combining check IDs with severities
checkov -d . --check CKV_AWS_1,HIGH,CRITICAL --hard-fail-on HIGH
# Result: Only CKV_AWS_1 processed, severities silently ignored

# Attempt 4: Different severity syntax variations
checkov -d . --check "HIGH,CRITICAL"
checkov -d . --check HIGH CRITICAL
checkov -d . --check high,critical
# Result: All silently ignored, no indication of the problem

What SHOULD happen (proposed solution):

$ checkov -d . --hard-fail-on HIGH,CRITICAL

⚠️  WARNING: Severity codes cannot be used without an API key.
Parameters --hard-fail-on contain severity codes: CRITICAL, HIGH
These parameters will be ignored during this scan.
Configure an API key with --bc-api-key to use severity-based filtering.

# Scan continues with check IDs only...

Expected behavior with API key (working correctly):

$ checkov -d . --hard-fail-on HIGH,CRITICAL --bc-api-key <key>
# No warning, severity filtering works as expected

Version:

  • Checkov Version: 3.2.495 (but issue exists in earlier versions)
  • Python Version: 3.12
  • Context: CI/CD pipeline standardization across multiple projects

Additional context

Why this matters:

  1. DevOps Experience: When building standardized pipelines, silent failures are the worst kind of bug. You think your security controls are working, but they're not.

  2. Documentation gap: The documentation mentions severity codes but doesn't prominently explain the API key requirement for these features to function.

  3. Time waste: After spending hours on trial-and-error, I had to read through the source code to understand the dependency on Bridgecrew/Prisma Cloud API for severity information.

  4. User confusion: Parameters like --hard-fail-on and --soft-fail-on appear to be core CLI features, but their severity-based functionality is actually platform-dependent.

What makes this issue important:

  • Silent failures are dangerous in security tooling - users assume controls are working when they're not
  • Wasted developer time - I'm likely not the only one who's gone through this frustration
  • Better UX - A simple warning would have saved hours of debugging
  • CI/CD reliability - Pipelines need explicit feedback when configuration is ineffective

Proposed implementation:

I've already implemented this feature in a branch and tested it thoroughly. The solution:

  • Detects severity codes in filtering parameters
  • Issues clear warnings when API key is missing
  • Explains that parameters will be ignored
  • Supports all documented formats (including CSV)
  • Maintains backward compatibility (warning only, no behavior changes)
  • Comprehensive test coverage (13 unit tests)

This would prevent future users from experiencing the same trial-and-error frustration I went through.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions