Repository navigation
[android][clr] No peer certificates when executing System.Net.Http.Functional.Tests on Android emulator #124526
Description
Build Information
Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1297669
Build error leg or test failing: System.Net.Http.Functional.Tests
Runtime changes: c1cb909...fbf6be5
02-17 19:11:51.890 10934 25477 I DOTNET : [IGNORED] System.Net.Http.Functional.Tests.SocketsHttpHandler_DiagnosticsTest_Http11.SendAsync_OperationCanceledException_RecordsActivitiesWithCorrectErrorInfo
02-17 19:11:51.891 10934 25477 I DOTNET : System.Net.Http.Functional.Tests.SocketsHttpHandler_DiagnosticsTest_Http11 1.4314913 ms
02-17 19:11:51.891 10934 25477 I DOTNET :
02-17 19:11:51.891 10934 25477 I DOTNET : Test collection for System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async
02-17 19:11:51.893 10934 25477 D DOTNET : GetJNIEnv: Registering JNI thread detach. env ptr 0x7dcd5de753c0. Key: -2147483632
02-17 19:11:53.125 10934 25477 D DOTNET : DetachThreadFromJNI: Detaching thread from JNI
02-17 19:11:54.064 10934 10945 I unctional.Test: Background young concurrent copying GC freed 129004(5772KB) AllocSpace objects, 0(0B) LOS objects, 24% free, 14MB/18MB, paused 381.045ms total 425.702ms
02-17 19:11:54.900 10934 25377 D DOTNET : GetJNIEnv: Registering JNI thread detach. env ptr 0x7dcd5de75500. Key: -2147483632
02-17 19:11:54.900 10934 25377 W System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates
02-17 19:11:54.901 10934 25377 W System.err: at com.android.org.conscrypt.ActiveSession.checkPeerCertificatesPresent(ActiveSession.java:314)
02-17 19:11:54.901 10934 25377 W System.err: at com.android.org.conscrypt.ActiveSession.getPeerCertificates(ActiveSession.java:180)
02-17 19:11:54.901 10934 25377 W System.err: at com.android.org.conscrypt.ExternalSession.getPeerCertificates(ExternalSession.java:106)
02-17 19:11:54.901 10934 25377 W System.err: at com.android.org.conscrypt.Java7ExtendedSSLSession.getPeerCertificates(Java7ExtendedSSLSession.java:129)
02-17 19:11:54.901 10934 25377 W System.err: at com.android.org.conscrypt.Java7ExtendedSSLSession.getPeerCertificates(Java7ExtendedSSLSession.java:31)
02-17 19:11:55.010 10934 25485 I DOTNET : [PASS] System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async.TestLoopbackAsync
Error Message
Fill the error message using step by step known issues guidance.
{
"ErrorMessage": [
"System.Net.Http.Functional.Tests",
"System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates"
],
"BuildRetry": false,
"ExcludeConsoleLog": false
}Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1297669
Error message validated: [System.Net.Http.Functional.Tests System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/17/2026 8:00:42 PM UTC
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 72 | 428 | 493 |
Report
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 0 | 4 | 19 |
Activity
[-][android] No peer certificates when executing System.Net.Http.Functional.Tests on Android emulator[/-][+][android][clr] No peer certificates when executing System.Net.Http.Functional.Tests on Android emulator[/+]EgorBo commentedon Feb 23, 2026
@dotnet/ncl Can someone take a look at this? it has 365 hits per week on CI, all JIT PRs are always red because of it.
ManickaP commentedon Feb 24, 2026
This is android specific AFAICS, isn't it?
cc @simonrozsival
simonrozsival commentedon Feb 24, 2026
@ManickaP yes
wfurt commentedon Feb 24, 2026
we may need to disable the offending tests if real fix drags too long.....
vitek-karas commentedon Feb 24, 2026
Investigation Summary: Android Emulator CI Failure
Root Cause
The failures are caused by an Android Emulator version upgrade (36.2.x → 36.4.9) picked up during Helix VMSS re-imaging on Feb 14-15.
Timeline
Helix-Ubuntu-2204-2025-11-12) — emulator ≤36.2.12linux-ubuntu-disable-unattended-upgradesto Android queues) trigger VMSS re-imagingEvidence
Failed to open QEMU pipe 'qemud:network': Invalid argumenterrors vs ~1 in green builds. This is consistent with emulator 36.2.6+ changing the WiFi packets handler to go throughnetsimdinstead of QEMU pipes — API 29 system images still try the oldqemud:networkpipe, which no longer works properly.linux-ubuntu-android-emulator.sh) usessdkmanager ... "emulator"which always installs the latest version — so re-imaging picked up 36.4.9.Recommended Fix
Pin the Android emulator version in the
dotnet-helix-machinesrepo'slinux-ubuntu-android-emulator.shto 36.2.12 (last known-good stable from Oct 13, 2025), or to 36.3.10 (Dec 4, 2025 stable). This prevents future uncontrolled emulator upgrades from breaking CI.wfurt commentedon Feb 24, 2026
Do you know what package name we need to pin @vitek-karas ? Also, there was debate with @simonrozsival and @akoeplinger that we may switch to docker images. That would give you full control over all aspects. But I'm not sure if we are ready....but it would be nice.
vitek-karas commentedon Feb 24, 2026
I'm not convinced that pinning the version is the right solution - we already have way too many things which need "regular" manual updates - it adds too much cost to the maintenance. Historically I'm not aware of Android emulator updates causing problems - so if this is a one-off - I would personally prefer to keep it auto-updating and dealing with rare occasional breaks when they happen.
Similar for docker images - yes - it would give us full control - but it comes with the same maintenance cost.
wfurt commentedon Feb 24, 2026
Agreed. While pinning may not be best option I think it would be ok to do it for now to get the CI green. We should keep looking for better strategy. But it is not clear to me what should be done right now and who is expected to do it. Also is this something that can be fixed via emulator config? I don't think @dotnet/ncl has enough insight to the Android details.
2 remaining items
vitek-karas commentedon Feb 25, 2026
Updated Investigation Findings (Feb 25) - Performance Analysis
Key Discovery: This is a Performance Regression, Not a Connectivity Failure
Detailed analysis of the Helix console logs reveals the tests are running and passing in the "red" builds — they just run too slowly and timeout.
Red build (1295230) detailed breakdown:
ALL Android work items are slower, not just HTTP:
The general slowdown is ~1.5-1.6x across all tests.
System.Net.Http.Functional.Testsis hit hardest because it has the longest Mono initialization time, which suffers a 3.8x slowdown.Local Validation on Linux x86_64 with KVM
Set up a D2s_v3 Azure VM (Ubuntu 24.04, KVM) with both emulator versions (36.3.10 and 36.4.9):
-memory 3072 -wipe-data -partition-size 1550 -skip-adb-auth -detect-image-hang -restart-when-stalled): 36.4.9 took >180s while 36.3.10 booted fasterThe raw compute performance inside the emulator is the same, which suggests the slowdown may be in I/O-intensive operations (assembly loading, memory-mapped files) or emulator-internal overhead during boot/initialization.
Conclusion
The VMSS re-imaging (triggered by PR 57760 on Feb 14) caused all Android emulator pools to get the latest emulator version (36.4.9). This version introduces a ~1.5-1.6x general performance regression on the Helix machines, with a 3.8x regression specifically in Mono runtime initialization.
System.Net.Http.Functional.Testsis the canary — it was already the slowest test (427s in green builds) and the slowdown pushes it past the 30-minute timeout.Recommended fix: Pin the emulator version in
linux-ubuntu-android-emulator.shto 36.3.10 (build 14472402) and trigger a re-image. The download URL format ishttps://dl.google.com/android/repository/emulator-linux_x64-{build_id}.zip.vitek-karas commentedon Feb 25, 2026
@wfurt I'm still looking into it - if what copilot found is true (general slowdown) then this is probably more complicated than disabling a test - in which case pinning the emulator version might be the way to go for now.
EgorBo commentedon Mar 3, 2026
Is there any way we can disable these tests while it's being investigated? 907 hits
Disable System.Net.Http.Functional.Tests on Android CoreCLR (#125171)
Re-enable System.Net.Http.Functional.Tests on Android and increase ti…
Re-enable System.Net.Http.Functional.Tests on Android and increase ti…
Revert Android Helix timeout to 30min to verify tests pass without in…
[Android] Re-enable System.Net.Http.Functional.Tests (#125481)
[Android] Re-enable System.Net.Http.Functional.Tests (#125481)