buckets: make a copy of all event maps before pouring the event to a bucket

[blotus]

Fixes #4459.

Each bucket now gets its own copy of the two maps a scenario can write to. Enriched and Unmarshaled are not cloned: Unmarshaled is intended to be used only in parsers, and the various enrichment method cannot be called from a scenario.

Cost, per (event, bucket) pour — realistic Caddy JSON event, M4 Pro:

	ns/op	B/op	allocs/op
before (maps shared)	11	0	0
after	185	1000	6

End to end, a pour into 48 loaded scenarios with 3 matching goes from 418 to 439
allocs/op; the wall-clock difference is inside the noise of that benchmark.

As a side-effect, this also fixes a potential race conditions in the meta:

• Scenario A sets a new meta with evt.SetMeta
• Scenario B overflows
• The meta set by scenario A would appear in the meta of the alert generated by scenario B because the maps were shared between all buckets.

[github-actions]

@blotus: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@blotus: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.48%. Comparing base (479f223) to head (af91422).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4481      +/-   ##
==========================================
- Coverage   70.99%   70.48%   -0.51%     
==========================================
  Files         505      522      +17     
  Lines       34790    35432     +642     
==========================================
+ Hits        24698    24975     +277     
- Misses      10091    10456     +365     
  Partials        1        1              

Flag	Coverage Δ
bats	46.24% <100.00%> (+0.01%)	⬆️
unit-linux	45.52% <100.00%> (+<0.01%)	⬆️
unit-windows	33.96% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[YakGravity]

Ran into this on v1.8.1 too, so it's still in the current release. Same spot as the report from 20 Aug: the panic is on pour, inside the uniq processor, not on overflow.

fatal error: concurrent map read and map write

goroutine 34359 [running]:
reflect.mapaccess_faststr(...)
	runtime/map.go:236 +0x18
reflect.Value.MapIndex(...)
	reflect/map.go:154 +0xbb
github.com/expr-lang/expr/vm/runtime.Fetch(...)
	github.com/expr-lang/expr@v1.17.8/vm/runtime/runtime.go:63 +0x245
github.com/expr-lang/expr/vm.(*VM).Run(...)
	github.com/expr-lang/expr@v1.17.8/vm/vm.go:142 +0xd47
github.com/crowdsecurity/crowdsec/pkg/leakybucket.getElement(...)
	github.com/crowdsecurity/crowdsec/pkg/leakybucket/uniq.go:82 +0xfb
github.com/crowdsecurity/crowdsec/pkg/leakybucket.(*UniqProcessor).OnBucketPour(...)
	github.com/crowdsecurity/crowdsec/pkg/leakybucket/uniq.go:30 +0x11b
github.com/crowdsecurity/crowdsec/pkg/leakybucket.(*Leaky).LeakRoutine(...)
	github.com/crowdsecurity/crowdsec/pkg/leakybucket/bucket.go:166 +0x85d

Version was v1.8.1-909b5157 from the official docker image, build date 3 Sep. It happened during a scanner burst, a single source filling two http scenarios within a couple of seconds, and the crash came while the overflow was being processed. Nothing exotic in the setup.

The process did not come back on its own. The container runs with restart: unless-stopped, but docker made no attempt: RestartCount stayed at 0 and there is no restart line in the daemon log. So the LAPI stayed down, and with the traefik bouncer plugin in live mode that fails closed, meaning 403 on every route behind the proxy rather than only the one the scanner was hitting. That lasted a bit over 13 hours until it was noticed. Moving the plugin to stream mode keeps a crash from taking everything down, but new decisions still stop until crowdsec is back.

Adding the scope here in case it is useful for prioritising.

[nicheosala]

Tested this on v1.8.1 (aarch64, Arch Linux ARM) — the race is still present there, and this patch fixes it.

I hit the same crash in production: agent died after ~11h30 with fatal error: concurrent map read and map write, this time with the read coming from alertcontext.EvalAlertContextRules via NewAlert → EventToContext (overflows.go:371) rather than EventsFromQueue.

Reproducer: 87 Caddy JSON access-log lines from a single WordPress scanner (POST /wp-json/batch/v1 and encoding variants, all 403/404/405/308, ~8s), replayed one-shot with a -race build. Stock hub config (crowdsecurity/caddy, base-http-scenarios, http-cve, http_base context), default routine counts, no AppSec, no postoverflows. Each run takes ~4s. Log attached.

A/B, alternating both binaries in the same loop, 20 iterations each:

build	runs with a race report	overflows per run
v1.8.1	12/20	3
v1.8.1 + 761e851	0/20	3

Only variable is the cherry-pick of 761e851 onto the v1.8.1 tag (I did not include 60fe0aa, which only adds benchmarks). The constant overflow=3 on both sides confirms the copy happens after the filter evaluation, so evt.SetMeta in a filter still lands in the bucket that set it — crowdsecurity/http-cve-probing keeps triggering normally.

On the writer side, in case it helps close the analysis in #4459: it is always a scenario filter calling evt.SetMeta() during filter evaluation in PourItemToHolders (manager_run.go:225). Two stock hub scenarios do this — crowdsecurity/http-cve-probing and crowdsecurity/http-technology-probing. Across 20 pre-fix runs the detector reported 20 races with the same writer and three different readers:

reader	occurrences
UniqProcessor.OnBucketPour → getElement (uniq.go:82)	11
NewAlert → EventsFromQueue (overflows.go:225)	8
NewAlert → alertFormatSource → eventSources (overflows.go:108)	1

Plus alertcontext from the production crash, so four distinct readers in total. Copying at pour time covers all of them, which a defensive copy inside EventsFromQueue would not have.

One minor note: maps.Clone is shallow, so for Unmarshaled (a map[string]any) the nested values stay shared between buckets — with the Caddy datasource, that's the whole decoded JSON object. Nothing writes into those nested structures today, so it isn't a live bug, just something that could bite later.

Happy to re-run the A/B on a rebased branch if that would help.
burst.log

[blotus]

/kind fix
/area agent