Skip to content

chore(analytics): drop unused Amplitude events and sample the rest - #14606

Merged
dylanjeffers merged 2 commits into
mainfrom
chore/trim-analytics-noise
Sep 24, 2026
Merged

dylanjeffers merged 2 commits into
mainfrom
chore/trim-analytics-noise

Conversation

@dylanjeffers

@dylanjeffers dylanjeffers commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

The Amplitude proxy (gain2.audius.co, metrics-worker) caps us at 1.95M events a month, and it has been hitting that in about 4 days. This PR removes most of the events the clients send: anything that's a duplicate, automatic, from a bot, or not used by any recent chart. What's left, apart from the core events, goes out from 10% of devices.

Volume before (Amplitude stored totals, Aug 31 to Sep 4, 5 days, project 265616)

# Event Stored Status after this PR
1 Session Start 118,654 removed, replaced by a client user property
2 Page View 44,992 kept (core; also used by sign-up charts)
3 Playback: Play 38,170 kept (core)
4 Search: Search 23,636 removed
5 Embed: Open Player 23,015 removed
6 Favorite 20,708 kept (core)
7 Buffering Time 18,473 removed
8 Repost 17,618 kept (core)
9 Explore: Section View 11,997 removed
10 Follow 9,718 kept (core)
11 Playback: Pause 8,047 kept (core)
12 Link Click 7,340 removed
13 session_end 7,279 removed (SDK automatic tracking off)
14 Share 5,821 kept (core)
15 Track Upload: Track Uploading 4,094 removed
16 Message Unfurl: Track 3,966 removed
17 [Amplitude] Form Started / Submitted 3,798 removed (SDK automatic tracking off)
18 Create Account: Select Genre 3,786 kept (core, sign-up)
19 Feed: Change view 3,298 sampled at 10% (Chronological Feed dashboard)
20 Comments: Load More Comments 3,222 removed

Web plus embed is 314k of that, iOS 66k and Android 64k. Some events are sent but never stored, so they don't show up above: [Amplitude] Page Viewed, session_start (both deleted in the project) and $identify.

What changed

1. SDK automatic events off, and bots skipped

  • Web defaultTracking: it now names every option. Any option left out defaults to on (isTrackingEnabled in analytics-client-common), so { sessions: true } had also been sending page views, form events and file downloads. Attribution stays on.
  • Bots: crawler and navigator.webdriver traffic gets no analytics on web or embed. On Aug 1-3, one Singapore "Chrome 145" crawler sent 371k Session Starts from 347k new devices.
  • Identify dedupe: identify only goes out when the traits change, on web and mobile, and is resent weekly.

2. Session Start removed everywhere

The platform it carried (Android Web, iOS Web, MacOS, Windows, Linux, Desktop Web, iOS App, Android App, Embed) is now a client user property. It's set through identify only when it changes for a device. Amplitude already computes sessions from session ids.

3. Events removed (call site, enum entry and type)

The rule: an event is removed if it isn't core and isn't referenced by an Amplitude chart, dashboard or cohort that was viewed or edited in the last 12 months.

I listed all 642 charts, dashboards, cohorts and notebooks in project 265616, then read the definitions of the 33 charts and 7 dashboards used since 2025-09-24. None of the 11 cohorts or 3 notebooks had been used in that window.

  • Session and diagnostics: Session Start, App Error, App Unexpected Error, Buffering Time, Not Found Page, Chat Websocket Error, Modal Opened/Closed
  • Search and navigation: Search: Search, Tag Search, Result Select, Tag Click, Link Click, Explore: Section View, Section Click, Trending: Change view, Profile Page: Tab Click, Sort, the social and website link clicks, Shown Artist Recommendations, Track Page: Play More By This Artist, Visualizer: Open/Close
  • Embed: Open Player, Player Error, Open modal
  • Comments (all interaction events): create, update, delete, like, report, mute, pin, reply, mention, timestamp and link clicks, load more/new, sort, drawer open/close, overflow menus, notification toggles, Comments History, Recent Comments
  • Chat: Create Chat, Chat Blast CTA/Create/Message Viewed, Send Message: Failure, Delete Chat, Set Chat Category, Block User, Change Inbox Settings, Send Message Reaction, Message Unfurl: Track/Playlist, Chat Entry Point (see note)
  • Playlists and library: Playlist: Add To Playlist, Open/Start/Complete Create, Make Public, Open Edit From Sidebar, Play; Playlist Library: Reorder; Folder: Open/Submit/Cancel Edit
  • Tracks: Track Download (all 6), Track Edit (all 3), Collection Edit (both), Track Replace (all 3), Track Upload: Track Uploading, View Track page, the follow and token gated upload variants; Follow/Token Gated unlocks; Stem: Complete Upload, Delete; Delete; Remix: New Remix, CoSign, CoSign Indicator, Hide; Artist Pick: Select Track
  • Remix contests: Create, Update, Delete, Pick Winners Open, Finalize Winners, View, Enter, View Submissions
  • Settings and account: Change Theme, Resend Account Recovery, Complete Change Password, Log Out, Deactivate Account (all 4), Developer Apps (all 8), Authorized Apps (both), Manager Mode (all 5), Export Private Key (all 5), Notifications: Toggle Setting, Browser Push Notification, Account Health: Upload Cover Photo, Upload Profile Picture, Download Desktop
  • Wallet and rewards: Connect Wallet (all 5), Create User Bank (both), Rewards Claim: Opened/Request/Success, Rewards Claim All: Failure
  • Banners and prompts: Banner Artist Coins/Yak Coin/Trading Volume/TOS clicks, Rate CTA (all 3)
  • Mobile offline mode: everything except Offline Play (the migration and download lifecycle events)

The analytics-only source props on SearchTag, ExternalLink and UserGeneratedText (linkSource) are removed too.

4. Sampling

  • Core allowlist: CORE_ANALYTICS_EVENTS in @audius/common (models/AnalyticsSampling.ts), plus the mobile share-channel events.
  • What's core: Playback Play/Pause, Listen, Page View, and the sign-up, sign-in, create-account and OAuth flows. Also purchase, buy, withdraw, USDC, Stripe, coin swaps and Launchpad; the upload open/start/complete/success/failure funnel; Share and share channels (including the notification and first-upload Twitter shares); Favorite, Repost and Follow and their undos; notification opens and clicks (including push and comment notifications); chat sends (Send Message: Success, Chat Blast: Message Sent); Report User: Chat (the only record of an abuse report); Rewards Claim All request/success; Embed: Copy; and all Weekly Rotation events.
  • Everything else: sent only from devices where an FNV-1a hash of the Amplitude device id falls in the bottom 10%. It carries sampleRate: 0.1, so counts scale back up by dividing by sampleRate. What's still sampled after step 3 is only chart-backed: Play Queue (all 7), Feed: Change view, Error Page, Account Health: Meter Full, Folder: Delete, the Playlist Library folder moves, Offline Play, and Chat Entry Point.
  • Where it runs: once, in the shared track path: web services/analytics/amplitude.ts and mobile services/analytics.ts. Embed now only sends core events.

Why the proxy counts about 5.5x what Amplitude stores

  • Confirmed: the proxy counted events that Amplitude drops without billing: session_start and [Amplitude] Page Viewed. It also counted $identify on every account load. Together that's roughly 50-70k a day.
  • Inferred: the other ~330k a day. The proxy counts a batch before forwarding it, so every SDK resend counts again (after 400/413/429/5xx responses, network errors, and the localStorage queue being replayed on the next load). Amplitude removes duplicates by insert_id, so it bills only one copy. AudiusProject/metrics-worker#1 gives back events Amplitude doesn't accept, and now also drops the removed event types that older clients still send.

Estimate against the 1.95M cap

Per day now Per day after Per month after
Amplitude stored (billed) ~100k ~35k (core 35.0k + sampled 0.1k) ~1.05M
Proxy count, if the gap is retries of rejected batches (fixed by worker#1) ~488k ~34k events + ≤8k identify ~1.3M
Proxy count, if the gap is duplicates Amplitude accepts with a 200 ~488k ~34k × 4.8 + 8k ~5.2M
  • Billed volume ends up under 1.95M either way, with about 45% headroom. Kept core events stored 175k over the 5 days. The sampled events stored 4.7k and will keep 10% of that.
  • The proxy count ends up under 1.95M only if the unexplained gap is retries of rejected batches. After deploying metrics-worker#1, compare the counter against Amplitude's daily totals for a few days to find out.
  • Rollout timing: web doesn't auto-deploy (Production gate), and mobile needs an OTA dispatch. Until those ship, metrics-worker#1's denylist is what stops old clients from spending the budget.

Kept even though it's low value

Behavior changes to review

  • Sign-out: Log Out used to run through the analytics callback. It's now called directly on desktop web and in the mobile web SignOutModal, still before the redirect.
  • Modal sagas: store/ui/modals/sagas.ts now returns no sagas. trackModalOpened/trackModalClosed still get dispatched and now do nothing.
  • Wallet-connect errors: the catch blocks in the mobile wallet-connect sagas only sent an error event, so they're now empty.
  • Mobile Chat screen: the blast "viewed" AsyncStorage bookkeeping was only for the removed event, so it's gone.

Tests

  • Typecheck: tsc passes for common, web and mobile. The local sdk and spl dist were rebuilt, because they were stale.
  • Lint: eslint on all touched files has no errors. Two exhaustive-deps warnings in the remix-contest host were already there.
  • Web vitest: 33 files and 177 tests pass. That includes services/analytics/amplitude.test.ts: automatic events off, client property set once per device, core events always sent, about 10% of devices kept for non-core events with sampleRate, identify dedupe, and bots skipped.
  • Common vitest: 172 of 179 pass. The 7 failures are in store/upload/selectors.test.ts, which fails the same way on an untouched checkout.
  • Embed jest: passes.
  • Mobile jest: can't run in this worktree, because react-native can't be resolved from the copied node_modules.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 5321bb9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

dylanjeffers and others added 2 commits September 24, 2026 11:48
- Web: turn off the SDK's automatic session_start/session_end, page view,
  form and file download events. Everything left out of defaultTracking
  defaults to on, and session_start and [Amplitude] Page Viewed are deleted
  in the project, so they were sent and counted by the proxy but dropped.
- Web: send Session Start once per Amplitude session instead of every page
  load.
- Web and mobile: skip identify when the traits have not changed (resent
  weekly).
- Web and embed: no analytics for crawlers and automated browsers.
- Embed: drop the per-load Session Start; Embed: Open Player already fires
  on every load and now carries source.
- Mobile: only report Buffering Time for buffers of at least 1s.
- Explore: Section View once per section per app session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Remove Session Start everywhere. The platform it carried is now a
  `client` user property, set when it changes for a device (web, mobile,
  embed).
- Remove every event that is not core and is not used by an Amplitude
  chart or dashboard viewed or edited in the last 12 months, with its call
  sites, enum entry and type. Embed: Open Player and Embed: Player Error
  go too.
- Send non-core events from a fixed 10% of devices, picked by a hash of
  the Amplitude device id, with a `sampleRate` property. Core events are
  an explicit allowlist in @audius/common (CORE_ANALYTICS_EVENTS) and are
  always sent.
- Drop the analytics-only `source` props on SearchTag, ExternalLink and
  UserGeneratedText.

Chat Entry Point is still fired from ShareModal and ShareDrawer, which are
being changed in a separate PR; the metrics proxy drops it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dylanjeffers
dylanjeffers force-pushed the chore/trim-analytics-noise branch from 20abde5 to 5321bb9 Compare September 24, 2026 19:09
@dylanjeffers dylanjeffers changed the title chore(analytics): cut duplicate and automatic Amplitude events chore(analytics): drop unused Amplitude events and sample the rest Sep 24, 2026
@dylanjeffers
dylanjeffers merged commit 7b7655d into main Sep 24, 2026
17 checks passed
@dylanjeffers
dylanjeffers deleted the chore/trim-analytics-noise branch September 24, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant