CHORE: Improve PR performance sensitivity and measurement reliability

[Gaurav Sharma (bewithgaurav)]

Work Item / Issue Reference

ADO Work Item: Fixed AB#44819

---

Summary

Make PR Performance Reports more sensitive to consistent changes while reducing first-use and short-measurement noise.

• Add tentative small impact above 10% and large impact above 20%, retaining the per-workload 1 ms floor and 80% pair agreement at each tier.
• Keep inconsistent movement out of headlines, badges, and signal fingerprints. Preserve raw percentages and agreement in the expanded timing table.
• Warm steady-state tasks inside every fresh measurement process. Keep connection opening cold.
• Measure short read/query tasks in identical fixed repetition blocks on both revisions, retaining raw inner timings and normalizing phase diagnostics per original workload.
• Validate repetition and warmup identity while preserving retained single-pass artifacts. Label clean partial coverage incomplete.

Validation

164 profiler CI contract tests passed. Pinned formatting and changed-file lint checks passed.

Five fresh-process same-build pairs against local SQL Server 2022 covered all 22 tasks with zero signals, 0.63% median absolute paired movement, and 3.40% maximum movement. This is a local macOS control, not a claim that hosted Linux variance has been eliminated.

No runtime driver changes, additional dependencies, dedicated-runner configuration, or CI matrix expansion.

[github-actions]

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 SMALL IMPROVEMENTS 0 SMALL SLOWDOWNS 0 LARGE IMPROVEMENTS 0 LARGE SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task	Before	After	Paired change	Agreement	Result
Connection opening	11.276 ms	10.978 ms	+0.1%	0/5 beyond 10%	no signal
SELECT queries	0.733 ms	0.747 ms	-0.4%	0/5 beyond 10%	no signal
Row insertion	30.986 ms	31.142 ms	+0.5%	0/5 beyond 10%	no signal
Executemany inserts	137.645 ms	135.256 ms	-2.1%	0/5 beyond 10%	no signal
Fetch-all queries	116.390 ms	116.233 ms	+0.1%	1/5 beyond 10%	no signal
Row-by-row fetching	12.567 ms	12.601 ms	+1.1%	0/5 beyond 10%	no signal
Batched row fetching	112.754 ms	111.668 ms	-0.5%	0/5 beyond 10%	no signal
Transaction commit and rollback	99.509 ms	99.863 ms	-0.1%	1/5 beyond 10%	no signal
Arrow row fetching	70.103 ms	70.711 ms	+2.0%	0/5 beyond 10%	no signal
100,000-row insertion	407.133 ms	400.473 ms	-0.4%	0/5 beyond 10%	no signal
Row fetching in batches of 100	107.260 ms	107.343 ms	-0.2%	0/5 beyond 10%	no signal
Row fetching in batches of 10,000	113.221 ms	116.250 ms	-3.8%	1/5 beyond 10%	no signal
Repeated positional queries	29.809 ms	29.579 ms	-0.9%	1/5 beyond 10%	no signal
Repeated named-parameter queries	32.679 ms	32.440 ms	+0.8%	0/5 beyond 10%	no signal
Legacy 100,000-row insertion	307.622 ms	308.623 ms	-1.1%	1/5 beyond 10%	no signal
Insertion with explicit input sizes	446.804 ms	431.983 ms	-3.7%	1/5 beyond 10%	no signal
Joined aggregation queries	187.441 ms	191.652 ms	+2.2%	0/5 beyond 10%	no signal
Large joined-result fetching	174.935 ms	177.015 ms	+1.9%	0/5 beyond 10%	no signal
1.2-million-row fetching	3685.522 ms	3698.344 ms	-0.8%	0/5 beyond 10%	no signal
Common table expression queries	4.825 ms	4.746 ms	-0.1%	1/5 beyond 10%	no signal
256 KiB VARCHAR(MAX) / fetchall()	1.275 ms	1.276 ms	-1.5%	0/5 beyond 10%	no signal
10,000 scalar values / fetchval() (debug disabled)	100.226 ms	100.338 ms	+0.2%	0/5 beyond 10%	no signal

Unix / SQL Server 2025

Database task	Before	After	Paired change	Agreement	Result
Connection opening	99.669 ms	100.234 ms	+0.6%	0/5 beyond 10%	no signal
SELECT queries	0.819 ms	0.811 ms	-1.4%	1/5 beyond 10%	no signal
Row insertion	35.195 ms	35.236 ms	-0.4%	0/5 beyond 10%	no signal
Executemany inserts	153.369 ms	156.287 ms	+0.2%	0/5 beyond 10%	no signal
Fetch-all queries	121.760 ms	122.510 ms	+0.5%	0/5 beyond 10%	no signal
Row-by-row fetching	14.688 ms	14.597 ms	+0.0%	1/5 beyond 10%	no signal
Batched row fetching	120.104 ms	122.229 ms	+3.0%	1/5 beyond 10%	no signal
Transaction commit and rollback	117.698 ms	121.050 ms	+3.0%	0/5 beyond 10%	no signal
Arrow row fetching	75.444 ms	77.036 ms	+2.1%	0/5 beyond 10%	no signal
100,000-row insertion	476.257 ms	468.972 ms	-2.9%	1/5 beyond 10%	no signal
Row fetching in batches of 100	123.166 ms	124.826 ms	+1.3%	0/5 beyond 10%	no signal
Row fetching in batches of 10,000	125.206 ms	126.095 ms	+1.8%	1/5 beyond 10%	no signal
Repeated positional queries	34.541 ms	34.189 ms	-1.4%	1/5 beyond 10%	no signal
Repeated named-parameter queries	37.085 ms	36.886 ms	-0.5%	0/5 beyond 10%	no signal
Legacy 100,000-row insertion	360.877 ms	377.192 ms	+3.8%	1/5 beyond 10%	no signal
Insertion with explicit input sizes	495.137 ms	506.315 ms	+2.3%	0/5 beyond 10%	no signal
Joined aggregation queries	162.983 ms	161.319 ms	-0.4%	2/5 beyond 10%	no signal
Large joined-result fetching	185.196 ms	191.426 ms	+5.1%	0/5 beyond 10%	no signal
1.2-million-row fetching	3674.194 ms	3791.566 ms	+0.2%	0/5 beyond 10%	no signal
Common table expression queries	4.777 ms	4.876 ms	+2.0%	0/5 beyond 10%	no signal
256 KiB VARCHAR(MAX) / fetchall()	1.455 ms	1.461 ms	+0.6%	0/5 beyond 10%	no signal
10,000 scalar values / fetchval() (debug disabled)	117.178 ms	108.920 ms	-7.3%	0/5 beyond 10%	no signal

Build and measurement details

ADO build 182814

PR head: 540e463c9963fdeeae759d1407e0a2adedfd62f8
Base: 9424705251baf8ded41a8a4968216b1fd9b00929
Measured merge: 85e9fa087a5289226ddc3cdccfd95b17269f8856

• Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4305.5; 5 paired comparisons and 1 discarded warmup pair.
  Steady-state tasks warm up once inside each worker; connection opening remains cold. Short-task blocks: SELECT queries: 100 repetitions; Row-by-row fetching: 10 repetitions; Repeated positional queries: 3 repetitions; Repeated named-parameter queries: 3 repetitions; Common table expression queries: 20 repetitions; 256 KiB VARCHAR(MAX) / fetchall(): 100 repetitions.
• Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5015.4; 5 paired comparisons and 1 discarded warmup pair.
  Steady-state tasks warm up once inside each worker; connection opening remains cold. Short-task blocks: SELECT queries: 100 repetitions; Row-by-row fetching: 10 repetitions; Repeated positional queries: 3 repetitions; Repeated named-parameter queries: 3 repetitions; Common table expression queries: 20 repetitions; 256 KiB VARCHAR(MAX) / fetchall(): 100 repetitions.

Tentative small impact requires more than 10% median paired movement; large impact requires more than 20%. Both require at least 1 ms between the median per-workload runtimes and at least 80% of pairs exceeding that tier's threshold in the same direction. Large impact is evaluated first. Unconfirmed movement remains no signal.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes. Repeated blocks use the mean time per original workload; phase timings and call counts are normalized per repetition. Inner repetitions are not additional independent pairs.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

[Copilot]

🔵 Needs a closer look

Hosted Linux behavior and timing reliability remain dependent on CI evidence beyond the supplied local macOS control.

0 open findings

What changed in this PR

Improves profiler sensitivity and reliability through tiered signals, warmups, repeated short workloads, and stricter artifact validation.

Changes:

• Adds 10% tentative-small and 20% large impact tiers.
• Warms steady-state scenarios and repeats short workloads.
• Normalizes diagnostics and expands tests/documentation.

File	Description
tests/​test_036_profiler_ci.py	Covers tiers, repetition validation, warmups, and reporting.
eng/​profiler_benchmarks/​workloads.py	Defines fixed short-workload repetition counts.
eng/​profiler_benchmarks/​report.py	Validates, classifies, normalizes, and renders results.
eng/​profiler_benchmarks/​README.md	Documents updated measurement semantics.
eng/​profiler_benchmarks/​controller.py	Implements per-process warmup and repeated measurement.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 85%

📈 Total Lines Covered: 9443 out of 11092 📁 Project: mssql-python

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)

mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 62.6%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection_pool.cpp: 82.3%
mssql_python.pybind.connection.connection.cpp: 83.1%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.fetch_temporal.hpp: 92.1%
mssql_python.cursor.py: 92.5%

---

🔗 Quick Links

⚙️ Build Summary	📋 Coverage Details
View Azure DevOps Build	Browse Full Coverage Report