tls: preserve CN name constraints with OpenSSL 4.1

[panva]

I'm opening this after checking out the OpenSSL 4.1 alpha release and having an LLM find behaviour changes, this one stood out as something that might need ironing out. Other 4.1 related test or doc fixes are in #65956.

I would much rather that someone more aware of TLS takes a look at this instead.

OpenSSL 4.1 no longer applies DNS name constraints to the subject CN by default, but tls.checkServerIdentity() still uses it without a DNS SAN. Enable subject checking during chain verification only when no DNS SAN exists, preserving the existing hostname verification policy.

Assisted-by: Codex

Refs: https://openssl-library.org/post/2026-09-09-openssl-4.1-alpha/

[nodejs-github-bot]

Review requested:

• @nodejs/crypto

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.17%. Comparing base (b805fb5) to head (f94d82b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65957      +/-   ##
==========================================
- Coverage   90.17%   90.17%   -0.01%     
==========================================
  Files         771      771              
  Lines      265489   265489              
  Branches    50471    50478       +7     
==========================================
- Hits       239405   239394      -11     
- Misses      17042    17048       +6     
- Partials     9042     9047       +5     

Files with missing lines	Coverage Δ
src/crypto/crypto_tls.cc	78.76% <ø> (ø)

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.