Repository navigation
az CLI fails with SSL certificate verification error on this dev machine — blocks all Azure resource creation #32
Description
Activity
Root cause confirmed
$ echo | openssl s_client -connect login.microsoftonline.com:443 -servername login.microsoftonline.com | openssl x509 -noout -issuer -subject issuer=OU=generated by Norton Antivirus for SSL/TLS scanning, O=Norton Web/Mail Shield, CN=Norton Web/Mail Shield Root subject=C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=stamp2.login.microsoftonline.comThis is Norton Antivirus's own SSL/TLS scanning feature — a legitimate, already-installed security product on this machine, not a malicious MITM. Norton intercepts HTTPS to scan traffic and re-signs it with its own certificate, injecting its root CA (
Norton Web/Mail Shield Root) into the Windows trusted root store. Confirmed present there viacertutil -store Root. Windows-native apps (browsers using the OS cert store) trust this fine; Python'scertifibundle (used by theazCLI) is a separate, vendored list that has no knowledge of it.What I tried
-
pip-system-certs(makes Python'ssslmodule defer to the OS trust store instead ofcertifi's bundled list) — the general, correct fix for exactly this class of problem, and it doesn't add any new trust, only aligns Python with certificates Windows (and by extension, this user) already trusts. Installed successfully into the system Python, but theazCLI ships its own isolated Python interpreter (C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe, invoked with-I, which ignores site-packages/env-based Python config by design) — so the system-Python fix doesn't reach it. Installing directly into az CLI's ownsite-packagesrequires admin rights to write underProgram Files, which this session doesn't have. -
Exported the Windows Root store to a PEM bundle (93 certs, includes Norton's), combined it with
certifi's bundle, and pointedREQUESTS_CA_BUNDLEat it for theazinvocation — this env var isn't blocked by-Iisolation (it's read by therequestslibrary, not Python's interpreter-level isolation). This got further: the "unable to get local issuer certificate" error went away, replaced by a new one:SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Basic Constraints of CA cert not marked criticalNorton's self-generated interception root CA doesn't set the
Basic ConstraintsX.509 extension ascritical, which is technically non-compliant with strict CA certificate requirements — OpenSSL rejects it as an intermediate/leaf-signing CA even once it's in the trusted bundle. This is a defect in how Norton generated its own certificate, not something fixable by adjusting what's trusted.
Where this leaves things
Getting past this now requires either:
- Disabling or excluding Azure domains from Norton's SSL/TLS scanning (a Norton settings change, not something I can or should do from here), or
- Relaxing certificate validation strictness in some way — which I'm not doing unilaterally; that's a real security-relevant tradeoff for you to decide, not me.
Recommend: open Norton's settings and either disable "SSL scanning" / "Secure VPN traffic inspection" entirely, or check whether it supports excluding specific domains/apps (
az,*.microsoftonline.com,*.azure.com) from inspection. Alternatively, run the Azure setup from a machine/network without Norton's SSL scanning enabled (e.g. Azure Cloud Shell in the browser, unaffected since it doesn't run a localazprocess).-
Follow-up: checked the remaining local-workaround options — none pan out
Investigated the four items raised as possible remaining paths. Summary: no new local fix. Cloud Shell remains the answer.
1. WSL2
wsl --statusshows the WSL platform itself is already enabled on this machine (Default Version: 2), butwsl -l -vreports zero installed distributions — there's no Linux userland to actually test Norton's interception against. Testing this option requireswsl --install <Distro>first, which is a real system change (installs a distro + downloads an image), so I didn't do it unilaterally.This is the one item worth revisiting: if you're willing to install a distro (e.g. Ubuntu), it's genuinely worth then testing
az/terraformfrom inside it — WSL2's NAT'd virtual network adapter may or may not sit behind Norton's interception layer, and that's not knowable without a distro present to test from. Flagging as your call, not doing it myself.2. Docker Desktop
Not installed — no
dockercommand, no Docker service, no install directory underProgram Files. Same reasoning as WSL2: untestable without installing it first, and I'm not installing it without checking with you.3. Newer az CLI version
Installed here:
2.87.0. Latest available:2.89.1. But this is not a promising path:- OS/system trust store support for
azCLI is an open, backlog-status feature request (Azure/azure-cli#19305, #28050) — no PR, no milestone, not shipped in any released version. - If anything, the trend is the opposite direction: Azure/azure-cli#32083, filed against 2.77.0, shows recent
azreleases tightening X.509 validation strictness (a different but analogous "CERTIFICATE_VERIFY_FAILED... Missing Authority Key Identifier" error), not relaxing it. The only workaround mentioned there isAZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1— a verification bypass, which is off the table here for the same reason--insecureis.
Upgrading
azCLI is not expected to resolve this and isn't worth doing on that basis.4. Norton fix for the "Basic Constraints not critical" defect
No documented fix, KB article, or acknowledgment found in Norton's support site or community forums for this specific defect in their SSL/TLS scanning root CA generation. Separately confirmed (read-only registry check, no settings touched) that this machine's Norton 360 is already on a current, recently auto-updated build (
26.7.11086.2615), alongside Norton Utilities Ultimate26.6.18812.9304and Norton AntiTrack4.8.7182.14364— so this isn't a stale-software gap; the defect is present in Norton's current release with no public sign they've fixed their own cert generation.Conclusion
No new viable local fix. Confirming explicitly: Norton's settings were not touched, and no TLS/certificate verification was weakened or bypassed during this investigation — only read-only checks (
wsl --status,wsl -l -v, Docker presence checks, registry reads,az version, public issue/doc research). Cloud Shell remains the recommended path for #31 until/unless a distro gets installed under WSL2 to test option 1, which is the only item left with any unexplored upside.- OS/system trust store support for
Status update — where this stands
Tried so far (see comments above for full detail):
pip-system-certson system Python — doesn't reachazCLI's isolated bundled Python (-Iflag ignores site-packages; installing into its actualsite-packagesunderProgram Filesneeds admin rights not available in this session).- Combined Windows Root store +
certifibundle viaREQUESTS_CA_BUNDLE— got past "unable to get local issuer certificate," but hit a deeper wall: Norton's self-generated root CA doesn't mark itsBasic Constraintsextension critical, which OpenSSL correctly rejects regardless of trust. Not fixable by adding more trust. - Investigated WSL2 (enabled, no distro installed — untested), Docker (not installed), newer
azCLI version (2.89.1 vs installed 2.87.0 — OS-trust-store support confirmed still unshipped upstream, wouldn't help), and a Norton-side fix for the cert defect (no known fix, Norton already on current build). None panned out.
Recommended fix, decided on together: a Norton exclusion (SSL/TLS scanning or Firewall Program Control) for the specific processes hitting this:
C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe(whatazactually runs)- Domains, if Norton supports domain-level exclusions instead:
login.microsoftonline.com,management.azure.com,*.azure.com
Where we're stuck: this requires changing Norton's configuration, which I'm not able to do myself (modifying system/security settings is outside what I can perform, even with explicit direction — it needs to happen in Norton's own UI, by the machine's owner). Owner needs to add the exclusion manually; exact menu path varies by Norton 360 version.
Next step once the exclusion is added: re-run the verification checks (
openssl s_clienthandshake againstlogin.microsoftonline.com,az account show,az group show) to confirm it actually resolves the issue before touching #31.Steps to resolve — for the machine owner
I can't make this change myself (modifying security software settings isn't something I can do, even on request) — here's exactly what to do.
1. Open Norton 360's settings
Open Norton from the system tray or Start menu → Settings (gear icon).
2. Find the SSL/TLS scanning or program-exclusion setting
The exact menu path varies by Norton 360 version — look for one of these (try Norton's in-app settings search box for "SSL" or "exclusion" if you don't see them right away):
- Firewall → Program Control — lets you set a per-program rule; look for an option to exclude a specific program from traffic inspection/scanning
- Antivirus → Scans and Risks → Exclusions / Items to Exclude from Scans — lets you add a specific
.exeto a trusted/excluded list
Per-program exclusion is preferable to any global "disable SSL scanning" toggle — it's narrower and doesn't drop protection for everything else.
3. Add this process to the exclusion
C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exeThis is what
azactually runs under the hood (az.cmdatC:\Program Files\Microsoft SDKs\Azure\CLI2\wbin\az.cmdjust wraps it).4. If Norton only offers domain-based exclusions instead of per-program
Add these instead:
login.microsoftonline.com management.azure.com *.azure.com5. Tell me once it's done
I'll re-run the exact checks used to diagnose this (
openssl s_clienthandshake,az account show,az group show) to confirm it's actually fixed before we touch #31.Correction — "Firewall → Program Control" was wrong
Verified against Norton's own official documentation (Configuring Smart Firewall settings): Program Control governs network access permissions only (Allow/Block/Ask per app) — it has nothing to do with SSL/TLS scanning or certificate interception. Different feature. Sorry for sending you looking in the wrong place.
What's actually responsible
Norton's "Email Protection" feature. Confirmed via the certificate file path: Norton's own troubleshooting docs reference
wscert.deratC:/ProgramData/Norton/AntiVirus/as the Email Protection certificate — this matches exactly thewscert.pempath the #34 investigation found in this machine'sNODE_EXTRA_CA_CERTSenv var. Despite the name, it's evidently intercepting general HTTPS (port 443) traffic, not just email ports.No documented per-program exclusion exists
Checked Norton's official support docs and community forums — found no setting to exclude a specific program or domain from this scanning. A Norton community moderator states directly: "they don't offer you a way to STOP Norton using its own certificate unfortunately." (source)
Norton's own official troubleshooting article for this exact certificate error (Troubleshooting email client warnings about invalid server certificates) offers only:
- Import Norton's certificate into the app's trust store — a dead end here, already confirmed: Norton's cert is structurally rejected (missing critical Basic Constraints extension) regardless of whether it's trusted.
- Reinstall Norton — unlikely to fix a certificate-generation defect.
The one real toggle found — broader than we wanted
Settings → Features tab → Antivirus → Email Protection tab (alternatively: Settings → Antivirus → Scans and Risks tab → "Email antivirus scan" toggle) turns Email Protection scanning off entirely — not scoped to just
python.exe/terraform.exe. This is a bigger tradeoff than the narrow per-app exclusion we were hoping for: it affects whatever protection Email Protection provides more broadly, not just the two processes blocking this work.Decision needed
- Try the broader Email Protection toggle if you're comfortable with that wider tradeoff — I can re-verify immediately after.
- Contact Norton support directly (chat/phone) to ask if there's an undocumented exclusion path — docs and forums didn't surface one, but support may know something not publicly documented.
- Skip this entirely and use Cloud Shell for One-time cloud setup needed before infra/terraform-test and infra/terraform-staging can be applied #31 — the zero-compromise path that was already the fallback recommendation.
Sources:
Decision: declined the Email Protection toggle — Norton stays untouched, full stop. No further local fix is being pursued for this issue. Path forward for the underlying work (#31) is Azure Cloud Shell, which sidesteps this machine entirely.
Final summary — all actions taken
Problem:
azCLI fails on this dev machine withSSLError: certificate verify failed: unable to get local issuer certificate.Root cause confirmed: Norton 360's "Email Protection" feature intercepts HTTPS (including
login.microsoftonline.com) and re-signs it with its own generated root CA. That CA doesn't mark itsBasic Constraintsextension critical — a structural defect that OpenSSL correctly rejects regardless of whether it's trusted.Actions taken, in order:
- Installed
pip-system-certson system Python to defer to the Windows trust store — didn't reachazCLI's isolated bundled Python (-Iflag ignores site-packages); fixing that needs admin rights to write underProgram Files, not available here. - Exported the Windows Root store (93 certs, includes Norton's), combined with
certifi's bundle, setREQUESTS_CA_BUNDLE— resolved the "unknown issuer" error but exposed the deeper "Basic Constraints not critical" defect, which is unfixable via trust configuration. - Investigated WSL2 (platform enabled, no distro installed — left untested rather than installing one unprompted), Docker Desktop (not installed, left alone), a newer
azCLI version (2.89.1 vs installed 2.87.0 — researched upstream, OS-trust-store support confirmed still unshipped, wouldn't help), and whether Norton has a known fix for the cert defect (none found; Norton already on a current build). - Identified the actual responsible feature as Norton Email Protection (corrected initial wrong guidance pointing at Firewall → Program Control, which only governs network access, not SSL scanning).
- Searched Norton's official docs and community forums for a per-program or per-domain exclusion — none exists. Norton's own troubleshooting article for this exact error offers only certificate import (dead end, per Bump actions/checkout from 4 to 7 #2 above) or reinstalling Norton.
- Found one real, working toggle: Settings → Features → Antivirus → Email Protection tab, which disables Email Protection scanning entirely — broader than a scoped exclusion.
Decision: declined to change any Norton settings, including the broader toggle. Norton's configuration remains untouched throughout this entire investigation — no changes were made to it at any point.
Outcome: no local fix exists within the given constraints (don't touch Norton, don't weaken TLS verification). Closing as not planned — the path forward for the underlying work (test/staging cloud environment setup) is Azure Cloud Shell, which runs entirely outside this machine and isn't subject to this interception.
- Installed
Problem
Every
azCLI call from this dev machine that needs a fresh login token fails with:Confirmed with plain read-only calls (
az group show,az account showfor cached/basic info works, but calls needing a fresh token fail). Reproduces identically regardless of sandboxing — this is the local machine's network/certificate configuration, not tool-specific.Likely cause
The network path from this machine to
login.microsoftonline.comappears to go through a proxy or endpoint-security product that intercepts HTTPS with its own certificate. Python'scertifibundle (used by theazCLI) doesn't include that proxy's root CA, so TLS verification fails before authentication can even happen.Confirmed: no
REQUESTS_CA_BUNDLEor similar override is currently set;certifiis using its stock bundle (certifi.where()→ the defaultcacert.pem).Impact
Blocks any
azCLI action that needs a live token — including the one-time Azure setup in #31 (creating Terraform remote state storage fortest/staging, and eventuallyterraform init/applyfor those stacks, since theazurermprovider goes through the same auth path).Fix options
REQUESTS_CA_BUNDLEat a combined bundle that includes it.azcommands from Azure Cloud Shell (portal.azure.com) or another machine that isn't behind the intercepting proxy.Not attempted
Disabling TLS certificate verification (e.g.
--debugflags or connection-verification overrides) — that defeats the purpose of the check and isn't a call to make unilaterally on an authenticated cloud CLI session without knowing whether the intercepting proxy is actually trusted.