rubygems: Fix Gem::Request for PQC support, adding integration connection tests

[junaruga]

This PR is related to #9542.

Summary

Added PQC server/client connection integration tests. As test_pqc_ssl_client_cert_auth_connection failed with the following error due to hardcoded OpenSSL::PKey::RSA.new in
Gem::Request.configure_connection_for_https, fixed it to support ML-DSA ssl_client_cert.

Error: test_pqc_ssl_client_cert_auth_connection(TestGemRemoteFetcherLocalSSLServer): OpenSSL::PKey::PKeyError: incorrect pkey type: UNDEF
/home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/openssl/pkey.rb:394:in 'OpenSSL::PKey::RSA#initialize'
/home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/openssl/pkey.rb:394:in 'Class#new'
/home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/openssl/pkey.rb:394:in 'OpenSSL::PKey::RSA.new'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request.rb:64:in 'Gem::Request.configure_connection_for_https'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request/https_pool.rb:7:in 'Gem::Request::HTTPSPool#setup_connection'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request/http_pool.rb:43:in 'Gem::Request::HTTPPool#make_connection'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request/http_pool.rb:23:in 'Gem::Request::HTTPPool#checkout'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request.rb:136:in 'Gem::Request#connection_for'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request.rb:194:in 'Gem::Request#perform_request'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/request.rb:161:in 'Gem::Request#fetch'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/remote_fetcher.rb:326:in 'Gem::RemoteFetcher#request'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/remote_fetcher.rb:217:in 'Gem::RemoteFetcher#fetch_http'
/home/jaruga/var/git/ruby/rubygems/lib/rubygems/remote_fetcher.rb:261:in 'Gem::RemoteFetcher#fetch_path'
/home/jaruga/var/git/ruby/rubygems/test/rubygems/test_gem_remote_fetcher_local_ssl_server.rb:98:in 'block in TestGemRemoteFetcherLocalSSLServer#test_pqc_ssl_client_cert_auth_connection'
      95:       ":ssl_ca_cert: #{temp_ca_cert}\n" \
      96:       ":ssl_client_cert: #{temp_client_cert}\n"
      97:     ) do |fetcher|
  =>  98:       fetcher.fetch_path("https://localhost:#{ssl_server.addr[1]}/yaml")
      99:     end
     100:   end
     101:
/home/jaruga/var/git/ruby/rubygems/test/rubygems/test_gem_remote_fetcher_local_ssl_server.rb:174:in 'TestGemRemoteFetcherLocalSSLServer#with_configured_fetcher'
/home/jaruga/var/git/ruby/rubygems/test/rubygems/test_gem_remote_fetcher_local_ssl_server.rb:94:in 'TestGemRemoteFetcherLocalSSLServer#test_pqc_ssl_client_cert_auth_connection'

In test/rubygems/test_gem_remote_fetcher_local_ssl_server.rb, created new tests, test_pqc_ssl_connection and test_pqc_ssl_client_cert_auth_connection The start_ssl_server has 2 modes: :non_pqc (default) and :pqc. With the mode :pqc, start_ssl_server runs with the RubyGems single PQC server with ML-KEM (X25519MLKEM768) key exchange and ML-DSA-65 certification.

Selected X25519MLKEM768 because rubygems.org supports X25519MLKEM768 for now. Selected ML-DSA-65 because it is used and tested
https://github.com/ruby/openssl/blob/master/test/openssl/test_ssl.rb - test_pqc_sigalg

Created tool/create_mldsa65_certs.sh to create ML-DSA-65 cert files, test/rubygems/mldsa65_*.pem. It is inspired by tool/create_certs.sh to create RSA cert files, test/rubygems/*.pem. Note the 65 in ML-DSA-65 is not bit length. ML-DSA-65 is algorithm name.

Notes

CI cases using OpenSSL >= 3.5 supporting PQC

Seeing the `.github/workflows/rubygems.yml, ubuntu-24.04 uses OpenSSL 3.0.13 which doesn't support PQC.

https://github.com/actions/runner-images/blob/ubuntu24/20260607.184/images/ubuntu/Ubuntu2404-Readme.md

OpenSSL 3.0.13-0ubuntu3.9

However, macos-26 uses OpenSSL 3.6.2 which supports PQC. The new PQC should run on the macos-26 cases.

https://github.com/actions/runner-images/blob/macos-26-arm64/20260525.0107/images/macos/macos-26-arm64-Readme.md

OpenSSL 3.6.2 7 Apr 2026 (Library: OpenSSL 3.6.2 7 Apr 2026)

Missing PQC dual server case

I wanted to add the mode - :pqc_dual to test with RubyGems dual certificate server. The code is as follows. However, in the case, RubyGems client needs a feature to control the signature algorithm such as OpenSSL::SSL::SSLContext#sigalgs used in https://github.com/ruby/openssl/blob/master/test/openssl/test_ssl.rb - test_pqc_sigalg. Because without this feature, the priority of selected signature algorithm depends on OpenSSL implementation. It can be unstable test.

It is possible to implement this feature by OPENSSL_CONF environment variable, and custom openssl.conf on a sub process using assert_separately to control the client's signature algorithm. but the logic is too complicated. So, I didn't add the PQC dual server test case. It is premature to add the case. Possibly we can implement the #sigalgs to the ruby/net-http first, then fetcher (Gem::RemoteFetcher) can implement #sigalgs calling ruby/net-http's #sigalgs.

  def start_ssl_server(config = {})
    ...
    case mode
    ...
    when :pqc_dual
      ctx.sigalgs = "rsa_pss_rsae_sha256:mldsa65"
      ctx.add_certificate(cert("mldsa65_ssl_cert.pem"), key("mldsa65_ssl_key.pem"))
      ctx.add_certificate(cert("ssl_cert.pem"), key("ssl_key.pem"))
      store = OpenSSL::X509::Store.new
      store.add_file File.join(__dir__, "mldsa65_ca_cert.pem")
      store.add_file File.join(__dir__, "ca_cert.pem")
      ctx.cert_store = store
      ctx.groups = "X25519MLKEM768"
    end
  end

What was the end-user or developer problem that led to this PR?

Gem::Request.configure_connection_for_https doesn't work with ML-DSA ssl_client_cert due to hardcoded OpenSSL::PKey::RSA.new.

What is your fix for the problem, implemented in this PR?

Fixed the Gem::Request.configure_connection_for_https to support ML-DSA ssl_client_cert.
Added integration HTTPS server/client connection tests.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[junaruga]

@hsbt and @rhenium, could you review this PR, as the PR heavily related to Ruby OpenSSL and OpenSSL? I need to fix the failing CI cases first. Thanks.

I confirmed the following command passed on my local.

$ bin/test-unit test/rubygems/test_gem_remote_fetcher_local_ssl_server.rb -v
Loaded suite Gem::TestCase
Started
TestGemRemoteFetcherLocalSSLServer:
  test_do_not_allow_insecure_ssl_connection_by_default:			.: (0.542613)
  test_do_not_allow_invalid_client_cert_auth_connection:		.: (0.525273)
  test_do_not_follow_insecure_redirect:					.: (0.535214)
  test_nil_ca_cert:							.: (0.524976)
  test_pqc_ssl_client_cert_auth_connection:				.: (0.562795)
  test_pqc_ssl_connection:						.: (0.532815)
  test_ssl_client_cert_auth_connection:					.: (0.546681)
  test_ssl_connection:							.: (0.529042)
  test_ssl_connection_allow_verify_none:				.: (0.522110)

Finished in 4.826097532 seconds.
---------------------------------------------------------------------------------------------
9 tests, 23 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
---------------------------------------------------------------------------------------------
1.86 tests/s, 4.77 assertions/s
Coverage report generated for rubygems to /home/jaruga/var/git/ruby/rubygems/coverage.
Line Coverage: 32.04% (1325 / 4135)

[junaruga]

Ah, I may find the cause of the failures in some CI cases. The start_ssl_server :pqc mode uses ctx.groups (OpenSSL::SSL::SSLContext#groups) which was renamed in the Ruby OpenSSL 4.0. In Ruby OpenSSL < 4.0, there is no #groups method.

[junaruga]

OK. After add the logic to omit PQC tests if Ruby OpenSSL < 4.0, all the tests passed.

[junaruga]

After rebasing the PR on the latest master branch, the following Bundler tests failed.

https://github.com/ruby/rubygems/actions/runs/27404697338/job/80990818846?pr=9615

rspec ./spec/install/gemfile/path_spec.rb:438 # bundle install with explicit source paths sets up executables

[junaruga]

OK. Now the PR's CI passed again after rebasing on the latest master branch.