diff --git a/AGENTS.md b/AGENTS.md index 3e969c33b..8e08c81d0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ This repository contains `fl-platform`, the Angular 22 standalone frontend for t - `src/app/services/`: auth/session, experiment orchestration, dashboard data access, algorithm rules, labeling, runtime env, theme, errors, and PDF/CSV exports. - `src/app/models/`: frontend and backend DTOs/interfaces for users, algorithms, experiments, filters, and data models. - `src/app/core/`: algorithm mapping, result enum mapping, constants, and result utility logic. -- `src/app/pages/experiment-studio/`: step views switched via a sticky horizontal stepper — `experiment-studio.component` shows one step view at a time (Datasets & Variables · Data review & preprocessing · Algorithm); all views stay mounted (hidden via CSS) so state is preserved. On the Data review step the stepper expands a sub-step row (Filtering, Raw Summary, Preprocessing, Processed Summary, Transformation) that drives the `statistic-analysis-panel` sections (`goToSection`). In-panel section targets use `scroll-margin-top` so `scrollIntoView` clears sticky chrome. Also contains variable/filter selection, QueryBuilder filter UI, algorithm configuration with variable role assignment (outcome y / covariates x on the algorithm panel), run/edit flows, statistics, visualizations, and result rendering. +- `src/app/pages/experiment-studio/`: step views switched via a sticky horizontal stepper — `experiment-studio.component` shows one step view at a time (Datasets & Variables · Data review & preprocessing · Algorithm); all views stay mounted (hidden via CSS) so state is preserved. On the Data review step the stepper expands a sub-step row (Filtering, Raw Summary, Preprocessing, Processed Summary, Transformation) that drives the `data-handling` (`DataHandlingComponent`) sections (`goToSection`). In-panel section targets use `scroll-margin-top` so `scrollIntoView` clears sticky chrome. Also contains variable/filter selection, QueryBuilder filter UI, algorithm configuration with variable role assignment (outcome y / covariates x on the algorithm panel), run/edit flows, statistics, visualizations, and result rendering. - `src/app/pages/experiments-dashboard/`: experiment list/search/pagination, detail view, compare mode, sharing, delete/edit/name updates, and result export. - `src/app/pages/terms-page/`: NDA/TOS display and acceptance flow. - `src/app/pages/account-page/`: account/profile view and logout entry. diff --git a/docs/context/conventions.md b/docs/context/conventions.md index ee621fc5c..5d0f70c67 100644 --- a/docs/context/conventions.md +++ b/docs/context/conventions.md @@ -30,7 +30,7 @@ ### UI and Styling - Use component CSS for feature-specific styles. - Use `src/styles.css` for app-wide styles and QueryBuilder theming. -- A component may list several stylesheets in `styleUrls`; Angular concatenates them **in array order**, so that order is the cascade. Splitting a file must preserve the original order and must not move a rule across another rule that shares its selector — `statistic-analysis-panel` is split this way (`.component.css` shell + pipeline canvas, then `.preprocessing.css`, then `.results.css`). +- A component may list several stylesheets in `styleUrls`; Angular concatenates them **in array order**, so that order is the cascade. Splitting a file must preserve the original order and must not move a rule across another rule that shares its selector — `data-handling` (`DataHandlingComponent`) is split this way (`.component.css` shell + pipeline canvas, then `.preprocessing.css`, then `.results.css`). Its section components attach to the element they replace (attribute selectors such as `div[app-pipeline-terminal]`) and `@import` the shared partials in `data-handling/styles/*.shared.css` at the top of their stylesheet (an `@import` must come first, so a rule only moves into a partial after checking it does not cross an equal-specificity rule for the same element); see the decision log entry on Data Handling sections. - Consult `DESIGN.md` before visual changes. - Existing branding assets live under `src/assets`. diff --git a/docs/context/decision-log.md b/docs/context/decision-log.md index 1e7e1f924..189722f4d 100644 --- a/docs/context/decision-log.md +++ b/docs/context/decision-log.md @@ -215,3 +215,26 @@ Consequences: Breaking: this UI needs a backend with PR #34 and does not work ag Files affected: `src/app/services/experiment-studio.service.ts`, `src/app/services/experiment-studio.service.spec.ts`, `e2e/experiment-studio-api.ts`, `e2e/experiment-studio.flows.spec.ts`, `README.md`, `docs/context/*.md`, `docs/llm-wiki/indexes/backend-endpoints.md`. Date: 2026-10-15. + +### Decision: Data Handling sections are attribute-hosted components over the pipeline's state + +Status: Accepted + +Context: `statistic-analysis-panel` grew to one ~4.5k-line component with a ~1.8k-line template. It was renamed `data-handling` (`DataHandlingComponent`, `app-data-handling`) and is being split into section components. Behaviour, visuals, `data-guide` anchors, e2e selectors, payloads and sessionStorage keys must not change, and the studio's global reveal rule (`.experiment-studio-page .workflow-section-body.open > *`) animates the direct children of a section body. + +Decision: +- A section component attaches to the element it replaces (`section[app-cohort-filter-station]`, `section[app-missing-values-step]`, `aside[app-prep-variable-rail]`, `div.transformation-create[app-categorical-column-card]`, `div.workflow-section-body[app-summary-workspace]`, ...) instead of adding a wrapper, so the DOM, the reveal animation and every selector stay as they were. `display: contents` wrappers were tried and rejected: they break the reveal animation. The host keeps the parent's `_ngcontent` attribute, so host-state rules are written `:host(.x) ...` in the child and ancestor rules `:host-context(.x) ...`. +- Each pipeline step is a station component (`source-data-station`, `cohort-filter-station`, `preprocessing-station`, `transformation-station`); the preprocessing station splits further into one component per sub-step plus one shared variable rail (`prep-variable-rail`, whose per-step row markup is given as ``), and each nominal transformation card's editor is `categorical-column-card`. +- Pipeline state and orchestration (requests, apply, persistence) stay on `DataHandlingComponent`, whose public API (`goToSection`, `progressStateChange`) and fields the studio and specs use. State that forms a feature lives in component-scoped services provided by the panel: `StatisticSummaryService` (the three summary surfaces, their cache keys, describe-response assembly), `PreprocessingStateService` (pending/applied rules of the three sub-steps and their comparisons), `TransformationDraftsService` (the nominal-column drafts, their status and payload), `SummaryExportService` (a surface's PDF report and CSV; the panel's template keeps the off-screen chart containers the PDF reads) and `SummaryHistogramService` (the Histogram tab's per-variable, per-split requests and cache; it reads the panel's blocks, pending preprocessing and Cohort Filtering preview through `ports`, and a settled request, with data or an error, is not repeated until its surface is rebuilt). The panel keeps its member names as getter/setter facades onto them, so templates and the studio are unchanged. +- Narrow sections take `input()`/`output()` (how-to strip, terminal, rail, card). Sections that read much of the pipeline inject `DATA_HANDLING` (`data-handling.context.ts`, provided by the panel with `useExisting`) and use `ChangeDetectionStrategy.Eager`, so they refresh exactly when the inline template did. +- `@ViewChild`/`@ViewChildren` lookups into extracted markup were replaced by registries the panel provides: `SectionAnchors` + `[dhSectionAnchor]` for scroll targets (it also runs the scroll request: `requestScroll` waits for the render that follows an open/close, and a newer request replaces one that has not run), `RuleBuilderRegistry` + `app-filter-config-modal[ruleBuilderFor]` for each category's rule builder (keyed by the rule object, not by position). +- Styles move with the elements that only a section renders. Rules several components render are `@import`ed partials in `data-handling/styles/` (`pipeline-node`, `statistics-panel`, `search-and-empty-states`, `tables`, `preprocessing-step`, `transformation-rules`), imported at the top of each user's sheet; keyframes travel with the rules that use them because emulated encapsulation scopes them per stylesheet. The panel's icon nudge (`.stats-panel-shell i`) is repeated as `:host-context(.stats-panel-shell) i`. +- Pure logic moves to `data-handling/utils/*` with specs (formatting, summary pivot and view helpers, preprocessing state, documentation HTML, transformation drafts, histogram preview parsing, outlier bounds, summary coverage/distribution/category reads over a pivot block, the section layout `goToSection` applies, export helpers); shared types live in `data-handling.models.ts`. + +Consequences: Each extraction up to the master merge was checked against master with a local DOM + computed-style + screenshot comparison of the pipeline in eight scenarios. The later shell-slimming commits (coverage reads, histogram loop, export, section layout and scroll request) are covered by their own specs, the panel's existing specs and the CI Playwright run, but were not rerun through that harness. Every component stylesheet is under the 20 kB budget. The panel is still the orchestrator (~3.2k lines of TS, from ~4.1k): the three describe requests, apply/persist and hydrating the stored preprocessing, the editors' validation and request-config builders, the pipeline-terminal and rail sub-node views, and the stage add/remove side effects remain there. The histogram loop, the exports, the coverage reads and the section layout/scroll request moved out. What is left needs a redesign rather than ports: the describe requests and apply/persist share the `*DescribeSeq` supersession counters, the preprocessing rule maps and `emitProgressState`, so extracting them means moving the pending/applied rule state's writers together (`PreprocessingStateService` owns the maps, but about a dozen panel methods, 43 assignment sites, still write them directly). + +After merging master (ordered transformation steps, PCA columns): the transformation station renders one card per store step (`app-categorical-column-card`, `app-kmeans-cluster-source`, `app-pca-column-source`), and `TransformationDraftsService` owns what master kept on the panel: the nominal drafts keyed by store step id, card open state, titles and chips per creator, name clashes (`columnNameError`, `writeProblem`), removal rules and the chooser (`choose`). The panel keeps the side effects (store writes, change detection). `TransformationStatisticsService` owns the category counts (placeholder shapes, the single describe over all derived columns, K-means column views, invalidation); the panel hands it its pending preprocessing and labels through `ports`, and the stage's views read its signals. + +Files affected: `src/app/pages/experiment-studio/data-handling/**`, `src/app/pages/experiment-studio/experiment-studio.component.*`, `AGENTS.md`, `docs/context/module-index.md`, `docs/context/conventions.md`, `docs/context/risk-register.md`, `docs/context/decision-log.md`. + +Date: 2026-10-23. diff --git a/docs/context/module-index.md b/docs/context/module-index.md index bdc4f92d6..d6df8e3f4 100644 --- a/docs/context/module-index.md +++ b/docs/context/module-index.md @@ -42,7 +42,7 @@ Notes: Stored historical experiment payloads may depend on legacy algorithm keys ## `src/app/pages/experiment-studio` Purpose: Experiment composition workflow. -Key files: `experiment-studio.component.*`, `variables-panel/*`, `statistic-analysis-panel/*` (including `pca-column-source` / `kmeans-cluster-source`), `algorithm-panel/*`, `execution-panel/*` (result view + docked setup summary), `stepper/*`, `shared/*`, `guide/*`. +Key files: `experiment-studio.component.*`, `variables-panel/*`, `data-handling/*` (panel shell + one component per pipeline step: `source-data-station/`, `cohort-filter-station/`, `preprocessing-station/` (with `prep-variable-rail/`, `missing-values-step/`, `outlier-step/`, `longitudinal-step/`, `preprocessing-state.service.ts`), `transformation-station/` (with `categorical-column-card/`, `kmeans-cluster-source/`, `pca-column-source/`, `transformation-drafts.service.ts`, `transformation-statistics.service.ts`); shared `summary-workspace/`, `pipeline-terminal/`, `how-to-strip/`; `statistic-summary.service.ts`, `summary-histogram.service.ts`, `summary-export.service.ts`; seams `data-handling.context.ts`, `section-anchors.ts`, `transformation/rule-builder-registry.ts`; shared CSS partials in `styles/`; `data-handling.models.ts`; pure `utils/`), `algorithm-panel/*`, `execution-panel/*` (result view + docked setup summary), `stepper/*`, `shared/*`, `guide/*`. Used by: `/experiment-studio` route and dashboard edit flows. Rules: Keep page-specific UI here; use `ExperimentStudioService` for shared selection/run state. Transformation cards append to `appliedTransformations` in creation order; each creator step names outputs with `new_column_names`. Do not reintroduce `pca_with_transformation`. Tests: Component specs live near components; e2e covers PCA then K-means then logistic regression payload order in `e2e/experiment-studio.flows.spec.ts`. diff --git a/docs/context/risk-register.md b/docs/context/risk-register.md index 14fff3e2c..9eb4907b5 100644 --- a/docs/context/risk-register.md +++ b/docs/context/risk-register.md @@ -15,8 +15,8 @@ | Docker image publishing | Release workflow uses registry credentials and pushes images. | `.github/workflows/publish_images.yml`. | Never print secrets; review registry/tag changes carefully. | Yes | | EBRAINS mirroring | Workflow pushes code/tags to an external GitLab mirror. | `.github/workflows/ebrains.yml`. | Avoid changing mirror refs or credentials handling without owner review. | Yes | | Dependencies | Updates can affect Angular build, bundle output, and chart/export libraries. | `package.json`, `package-lock.json`. | Explain need, review lockfile, run build/tests. | Yes | -| Cohort / category-rule filters (edit mode) | Stored filter trees can be re-sent in a shape the engine rejects, or silently rewritten, so an experiment that runs from the API fails from Studio. | `core/filter-logic.utils.ts`, `filter-config-modal.component.ts` (`backendToGroup`/`normalizeCondition`), `statistic-analysis-panel.component.ts` (`commitTransformationRuleFilters`). | Treat backend filter trees as untrusted input: hydrate every shape, never default an unknown operator to `equal`, never overwrite a loaded rule with an empty builder. Verify edit-mode runs against an experiment whose filters use `in`/`is_not_null`. | Yes | -| Derived-column stage (ungated) | Preview and Apply run on whatever the cards can express, so an unfinished card contributes nothing to the describe and to the payload, and a duplicate column name persists its first occurrence only. | `statistic-analysis-panel.component.ts` (`commitTransformationRuleFilters`, `buildTransformationConfigForDraft`, `transformationDraftStatus`). | Do not reintroduce a Preview/Apply gate over loaded data: it strands experiments whose category rules were authored outside this UI. Keep the commit's rule-safety check — never overwrite a rule its builder could not read — and let that rule's own note (`.transformation-saved-filter-note`, driven by `unloadedInput()`) plus the `Pending`/`Duplicate name` chips stay the only warning. `transformationCanApply` may dim Apply while no card forms a config, which must stay a statement about an empty payload and never a verdict on loaded rules; and Apply must keep the stage open while `transformationHasPendingChange` is true, or the chip that reports unfinished work folds away with it. Manual QA must cover Apply with a duplicate column name, and Apply on an experiment whose stored category rules this UI cannot author. | Yes | +| Cohort / category-rule filters (edit mode) | Stored filter trees can be re-sent in a shape the engine rejects, or silently rewritten, so an experiment that runs from the API fails from Studio. | `core/filter-logic.utils.ts`, `filter-config-modal.component.ts` (`backendToGroup`/`normalizeCondition`), `data-handling.component.ts` (`commitTransformationRuleFilters`). | Treat backend filter trees as untrusted input: hydrate every shape, never default an unknown operator to `equal`, never overwrite a loaded rule with an empty builder. Verify edit-mode runs against an experiment whose filters use `in`/`is_not_null`. | Yes | +| Derived-column stage (ungated) | Preview and Apply run on whatever the cards can express, so an unfinished card contributes nothing to the describe and to the payload, and a duplicate column name persists its first occurrence only. | `data-handling.component.ts` (`commitTransformationRuleFilters`), `transformation-station/transformation-drafts.service.ts` (`status`, `writeProblem`), `utils/transformation-draft.utils.ts` (`transformationConfigFor`). | Do not reintroduce a Preview/Apply gate over loaded data: it strands experiments whose category rules were authored outside this UI. Keep the commit's rule-safety check — never overwrite a rule its builder could not read — and let that rule's own note (`.transformation-saved-filter-note`, driven by `unloadedInput()`) plus the `Pending`/`Duplicate name` chips stay the only warning. `transformationCanApply` may dim Apply while no card forms a config, which must stay a statement about an empty payload and never a verdict on loaded rules; and Apply must keep the stage open while `transformationHasPendingChange` is true, or the chip that reports unfinished work folds away with it. Manual QA must cover Apply with a duplicate column name, and Apply on an experiment whose stored category rules this UI cannot author. | Yes | | Ordered transformations / PCA columns | Reordering or keying creators by name can break "later step may only read earlier outputs"; sending `code` or `pca_with_transformation` fails at Exaflow. | `ExperimentStudioService.appliedTransformations`, `pca_column_creator` / `kmeans_cluster_creator` / `categorical_column_creator`, e2e PCA→K-means→logistic flow. | Keep creators in the ordered list with `new_column_names`; emit after singletons; never revive `pca_with_transformation`. Manual QA: two PCA cards, K-means on PCA outputs, logistic on outputs, edit/rerun. | Yes | | Dashboard history cap | A filter the backend cannot express is answered from one ownership-scoped snapshot of the history, capped at `fullHistoryCap` runs; a search past it silently misses older runs. | `experiments-dashboard.service.ts` (`loadAllExperiments`, `historyTruncated`). | The cap is stated in the list summary now, not hidden; removing the signal without a backend substring/date filter brings the silent miss back. | Yes | | Design and branding | UI changes can violate MIP brand guidelines. | `DESIGN.md`, logos/assets. | Follow approved colors, logo rules, and UI hierarchy. | For visible brand changes | diff --git a/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.css b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.css new file mode 100644 index 000000000..77756c9e9 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.css @@ -0,0 +1,31 @@ +@import '../styles/pipeline-node.shared.css'; + +/* The panel nudges every icon 1px down (`.stats-panel-shell i`); the rule is + scoped to the panel's own template, so it is repeated here for this one. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} + +/* The panel's `.pipeline-node .workflow-section-body`, for this step's bodies. */ +:host(.pipeline-node) .workflow-section-body { + padding-inline: 16px; +} + +/* Live match chip under the cohort builder, filled by Preview. */ +.filter-match-row { + display: flex; + justify-content: flex-end; + padding: 14px 14px 10px; +} + +.filter-match-chip { + display: inline-flex; + align-items: center; + gap: 8px; + padding: 4px 12px; + border-radius: 999px; + font-size: 0.78rem; + font-weight: 600; + color: #065f46; + background: color-mix(in srgb, var(--success) 12%, white); +} diff --git a/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.html b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.html new file mode 100644 index 000000000..4db13ad04 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.html @@ -0,0 +1,110 @@ +
+
+
+
+Step 2 · Optional +

+ Keep only the patient records you want — for example, age 60 and over. +
+
+
+ @if (dh.filterRuleCount() > 0 && dh.recordsOut.filtered !== null) { + Records out{{ dh.recordsOut.filtered!.toLocaleString() }}@if (dh.recordsOut.source !== null) { of {{ dh.recordsOut.source!.toLocaleString() }}} + } + + {{ dh.filteringStatusLabel }} + + @if (dh.isStepAdded('filters')) { + + } + +
+
+ + +@if (!dh.sectionOpen().filters && !dh.sectionOpen().raw) { +
+ @if (dh.cohortRuleRows.length) { +

Records are kept only if they match {{ dh.cohortMatchesAny ? 'any of these rules' : 'these rules' }}. Click a rule to edit it.

+ } +
+ @for (row of dh.cohortRuleRows; track $index) { +
+ + +
+ } +
+ + +
+
+
+} + +@if (dh.isStepAdded('filters')) { +
+ + @if (dh.sectionOpen().filters) { +
+ } +
+ + + + @if (dh.filterPreviewStatus) { +
+ {{ dh.filterPreviewStatus }} +
+ } + + +
+
+
+
+
+} diff --git a/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.spec.ts b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.spec.ts new file mode 100644 index 000000000..6fd88e504 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.spec.ts @@ -0,0 +1,64 @@ +import { Component, signal } from '@angular/core'; +import { TestBed } from '@angular/core/testing'; +import { vi } from 'vitest'; +import type { DataHandlingComponent } from '../data-handling.component'; +import { DATA_HANDLING } from '../data-handling.context'; +import { SectionAnchors } from '../section-anchors'; +import { CohortFilterStationComponent } from './cohort-filter-station.component'; + +@Component({ + imports: [CohortFilterStationComponent], + template: `
`, +}) +class HostComponent {} + +/** The folded step: rules exist but the builder is not open, so no editor is mounted. */ +function render(rows: Array<{ title: string; subtitle: string }>, matchesAny = false) { + const dh = { + recordsOut: { filtered: 80, source: 100 }, + filterRuleCount: signal(rows.length), + filteringStatusLabel: rows.length ? 'Applied' : 'Not in run yet', + sectionOpen: signal({ filters: false, raw: false }), + cohortRuleRows: rows, + cohortMatchesAny: matchesAny, + isStepAdded: () => false, + isStepOpen: () => false, + onStepHeaderClick: vi.fn(), + toggleStep: vi.fn(), + goToSection: vi.fn(), + removeStep: vi.fn(), + }; + TestBed.configureTestingModule({ + providers: [{ provide: DATA_HANDLING, useValue: dh as unknown as DataHandlingComponent }, SectionAnchors], + }); + const fixture = TestBed.createComponent(HostComponent); + fixture.detectChanges(); + return { dh, el: fixture.nativeElement as HTMLElement }; +} + +describe('CohortFilterStationComponent', () => { + it('lists the stored rules in the folded overview, each leading back to the builder', () => { + const { dh, el } = render([{ title: 'Age', subtitle: 'is at least 60' }], true); + expect(el.querySelector('.pipeline-subnodes-caption')?.textContent).toContain('any of these rules'); + const items = el.querySelectorAll('[aria-label="Cohort filtering rules"] .pipeline-subnode-item'); + expect(items.length).toBe(2); + expect(items[0].textContent).toContain('Age'); + (items[0].querySelector('button') as HTMLButtonElement).click(); + expect(dh.goToSection).toHaveBeenCalledWith('filters'); + }); + + it('shows only the Add condition rail when there are no rules', () => { + const { el } = render([]); + expect(el.querySelector('.pipeline-subnodes-caption')).toBeNull(); + expect(el.querySelector('.pipeline-subnode-item.is-ghost')?.textContent).toContain('Add condition'); + expect(el.querySelector('.pipeline-records-out')).toBeNull(); + }); + + it('mounts no rule builder while the step is not added', () => { + const { el } = render([{ title: 'Age', subtitle: '' }]); + expect(el.querySelector('app-filter-config-modal')).toBeNull(); + const value = el.querySelector('.pipeline-records-out-value')!; + expect(value.firstChild?.textContent).toBe('80'); + expect(value.querySelector('span')?.textContent).toBe('of 100'); + }); +}); diff --git a/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.ts b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.ts new file mode 100644 index 000000000..d4504e0f9 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/cohort-filter-station/cohort-filter-station.component.ts @@ -0,0 +1,25 @@ +import { ChangeDetectionStrategy, Component, inject } from '@angular/core'; +import { FilterConfigModalComponent } from '../../shared/filter-config-modal/filter-config-modal.component'; +import { HowToStripComponent } from '../how-to-strip/how-to-strip.component'; +import { SectionAnchorDirective } from '../section-anchors'; +import { StationActionBarComponent } from '../../shared/station-action-bar/station-action-bar.component'; +import { StationCardComponent } from '../../shared/station-card/station-card.component'; +import { SummaryWorkspaceComponent } from '../summary-workspace/summary-workspace.component'; +import { DATA_HANDLING } from '../data-handling.context'; + +/** + * Step 2 of the pipeline: optional cohort filtering (rule builder, preview, apply) and the Raw preview it feeds. + * Attached to the element it renders into, so the pipeline's DOM, anchors and selectors + * are unchanged; reads and drives the pipeline through DATA_HANDLING. + */ +@Component({ + selector: 'section[app-cohort-filter-station]', + imports: [StationCardComponent, StationActionBarComponent, FilterConfigModalComponent, HowToStripComponent, SummaryWorkspaceComponent, SectionAnchorDirective], + templateUrl: './cohort-filter-station.component.html', + styleUrl: './cohort-filter-station.component.css', + // Reads the pipeline's plain fields, so it refreshes with the pipeline (see DATA_HANDLING). + changeDetection: ChangeDetectionStrategy.Eager, +}) +export class CohortFilterStationComponent { + protected readonly dh = inject(DATA_HANDLING); +} diff --git a/src/app/pages/experiment-studio/data-handling/data-handling.component.css b/src/app/pages/experiment-studio/data-handling/data-handling.component.css new file mode 100644 index 000000000..456725c90 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/data-handling.component.css @@ -0,0 +1,137 @@ +@import './styles/statistics-panel.shared.css'; +@import './styles/pipeline-node.shared.css'; + +.pipeline-node { + scroll-margin-top: calc( + var(--header-height, 64px) + var(--studio-sub-header-height, 44px) + var(--warning-banner-space, 0px) + 16px + ); +} + +.stats-panel-shell { + --panel-success-bg: #dfefe4; + --panel-success-text: #166534; + --panel-warning-text: color-mix(in srgb, var(--primary-color) 70%, var(--accent-color) 30%); + + background: transparent; + display: flex; + flex-direction: column; + overflow: visible; + font-family: inherit; +} + +.modal-content { + display: flex; + flex-direction: column; + gap: 0; + background: transparent; + overflow: visible; +} + +.stats-panel-shell i { + transform: translateY(1px); +} + +/* ========================================================================= + VERTICAL NODE-PIPELINE CANVAS + ========================================================================= */ + +/* Width is explicit on purpose: `margin: 0 auto` opts this flex item out of the + cross-axis stretch it would otherwise get, so an auto width resolves to + max-content — the widest open section then set the width of every station + card (the summary table made the whole rail ~120px wider than Filtering). + One capped width keeps Filtering, both summaries and Preprocessing on the + same two edges. */ +.pipeline-canvas-shell { + display: flex; + flex-direction: column; + align-items: stretch; + width: min(100%, var(--studio-step-max, 1160px)); + max-width: 100%; + margin: 0 auto; + gap: 0; +} + +/* Every step renders as a solid card; one that adds nothing to the next run says so + in its status chip. A collapsed step (`is-compact`) shows only its overview rail, + so it reads as a compact flowchart at `--studio-pipeline-overview-max`. An open + step re-expands to the full rail width, which is what the QueryBuilder, summary + tables and charts need. `align-self` opts the row out of the shell's cross-axis + stretch. The terminal card (pipeline-terminal) repeats the cap on its host. */ +.pipeline-node.is-compact { + align-self: center; + width: 100%; + max-width: var(--studio-pipeline-overview-max, 860px); +} + +/* The header titles the whole rail rather than one step, so it is not a card: + one hairline under the title, no frame or shadow, and the stage chips sit on + that line instead of in a badge box. */ +.pipeline-canvas-header { + display: flex; + align-items: flex-end; + justify-content: space-between; + box-sizing: border-box; + width: 100%; + min-height: 0; + margin-bottom: 18px; + padding: 4px 2px 18px; + background: transparent; + border: 0; + border-bottom: 1px solid var(--studio-card-header-border); + border-radius: 0; + box-shadow: none; + gap: 16px; +} + +.pipeline-header-title-group h2 { + font-size: 1.15rem; + font-weight: 700; + color: var(--text-main); + margin: 0 0 4px 0; + letter-spacing: 0; +} + +.pipeline-header-title-group p { + font-size: 0.82rem; + color: var(--text-muted); + margin: 0; + line-height: 1.4; + max-width: 62ch; + text-wrap: pretty; +} + +/* Pipeline Node Card */ +.pipeline-node { + background: var(--card-bg, #ffffff); + border: 1px solid var(--studio-card-border); + border-radius: var(--radius-lg, 12px); + box-shadow: var(--studio-card-shadow); + transition: border-color 0.2s ease, box-shadow 0.2s ease, opacity 0.2s ease; + overflow: hidden; +} + +/* Step 0 is permanent, so it borrows the compact width of the collapsed rows + rather than the full rail; only its snapshot needs the wide pivot table, and + widening then keeps the card and the table on the same two edges. */ +.pipeline-node.is-readonly { + align-self: center; + width: 100%; + max-width: var(--studio-pipeline-overview-max, 860px); +} + +.pipeline-node.is-readonly.is-active { + align-self: stretch; + max-width: 100%; +} + +@media (max-width: 768px) { + .pipeline-canvas-header { + flex-wrap: wrap; + } +} + +/* Inside a pipeline node the station card draws its own frame; without an inset + it sat flush on the node border and the two 1px lines vibrated as one. */ +.pipeline-node .workflow-section-body { + padding-inline: 16px; +} diff --git a/src/app/pages/experiment-studio/data-handling/data-handling.component.html b/src/app/pages/experiment-studio/data-handling/data-handling.component.html new file mode 100644 index 000000000..ee1cf3a34 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/data-handling.component.html @@ -0,0 +1,122 @@ +
+
+
+
+
+

Data Handling Pipeline

+

Get your records ready for analysis. Stages run top to bottom — Raw data is a read-only preview, Preprocessing always runs, and Filtering and Transformations are optional.

+
+
+ + +
+ + + + + +
+ + + + + + +
+ + + + + +
+ + + + + +
+ + + @for (kind of summaryKinds; track kind) { + @if (getSummary(kind); as summary) { +
+
+ @for (v of summary.nonNominalVariables; track v.code; let i = $index) { + + } +
+
+ @for (v of summary.nominalVariables; track v.code; let i = $index) { + + } +
+
+ } + } +
+
+
+ +@if (isExporting) { +
+
+ + +
+
+} diff --git a/src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.spec.ts b/src/app/pages/experiment-studio/data-handling/data-handling.component.spec.ts similarity index 96% rename from src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.spec.ts rename to src/app/pages/experiment-studio/data-handling/data-handling.component.spec.ts index 141e69666..15a1c7053 100644 --- a/src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.spec.ts +++ b/src/app/pages/experiment-studio/data-handling/data-handling.component.spec.ts @@ -1,7 +1,7 @@ import type { MockedObject } from 'vitest'; import { ComponentFixture, TestBed } from '@angular/core/testing'; -import { QueryList } from '@angular/core'; -import { StatisticAnalysisPanelComponent } from './statistic-analysis-panel.component'; +import { DataHandlingComponent } from './data-handling.component'; +import { RuleBuilder } from './transformation/rule-builder-registry'; import { ExperimentStudioService } from '../../../services/experiment-studio.service'; import { ExperimentsDashboardService } from '../../../services/experiments-dashboard.service'; import { ExperimentStudioNavigationService } from '../../../services/experiment-studio-navigation.service'; @@ -13,9 +13,29 @@ import { provideZonelessChangeDetection, signal } from '@angular/core'; import { provideEchartsCore } from 'ngx-echarts'; import { createSpyObj } from '../../../testing/spy-obj.testing'; -describe('StatisticAnalysisPanelComponent', () => { - let component: StatisticAnalysisPanelComponent; - let fixture: ComponentFixture; + +/** A stand-in rule builder: what it would export, and the error it would show. */ +interface FakeRuleBuilder { + exportFilterLogic: () => unknown; + filterError: () => string | null; +} + +/** + * Stands in for the rendered category builders: `builders[i]` edits the i-th rule in + * card order; rules past the end of the list have no builder (as if not rendered). + */ +function useRuleBuilders(component: DataHandlingComponent, builders: readonly FakeRuleBuilder[]): void { + component.transformationDrafts.flatMap((draft) => draft.rules).forEach((rule, index) => { + const live = component.ruleBuilders.get(rule); + if (live) component.ruleBuilders.unregister(rule, live); + const fake = builders[index]; + if (fake) component.ruleBuilders.register(rule, { ensureFirstCondition: () => undefined, ...fake } as RuleBuilder); + }); +} + +describe('DataHandlingComponent', () => { + let component: DataHandlingComponent; + let fixture: ComponentFixture; let mockExpService: MockedObject; let mockChartBuilder: MockedObject; let mockPdfService: MockedObject; @@ -129,7 +149,7 @@ describe('StatisticAnalysisPanelComponent', () => { mockPdfService = createSpyObj('PdfExportService', ['exportDescriptiveStatisticsPdf']); await TestBed.configureTestingModule({ - imports: [StatisticAnalysisPanelComponent], // Standalone component + imports: [DataHandlingComponent], // Standalone component providers: [ provideZonelessChangeDetection(), provideEchartsCore({ @@ -142,7 +162,7 @@ describe('StatisticAnalysisPanelComponent', () => { ] }).compileComponents(); - fixture = TestBed.createComponent(StatisticAnalysisPanelComponent); + fixture = TestBed.createComponent(DataHandlingComponent); component = fixture.componentInstance; fixture.detectChanges(); }); @@ -491,7 +511,7 @@ describe('StatisticAnalysisPanelComponent', () => { configureRawSummary(); component.goToSection('transformation'); // The stage opens on its type chooser: the editor, and its preview, live in the card. - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); fixture.detectChanges(); const transformation = workflowSection('Transformation'); @@ -727,7 +747,7 @@ describe('StatisticAnalysisPanelComponent', () => { openStation('setup'); openStation('transformation'); // The stage opens on its type chooser, so a column card exists only once a type is chosen. - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); fixture.detectChanges(); // Filtering and Preprocessing keep a preview action. A categorical column does not: @@ -770,9 +790,9 @@ describe('StatisticAnalysisPanelComponent', () => { openStation('setup'); openStation('transformation'); component.enableOutlierHandling(); - component.chooseTransformation('kmeans'); + component.chooseTransformation('kmeans_cluster_creator'); // The column card joins the graph only after its type is chosen. - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); fixture.detectChanges(); const headers = Array.from((fixture.nativeElement as HTMLElement).querySelectorAll('.pipeline-node .station-card-header')); @@ -948,7 +968,7 @@ describe('StatisticAnalysisPanelComponent', () => { expect(component.isStepAdded('setup')).toBe(false); expect(mockExpService.loadDescriptiveOverview).toHaveBeenCalled(); expect(component.preprocessingStatus).toBe('none'); - expect((component as any).userPreprocessingApplied).toBe(false); + expect(component.preprocessingState.userApplied).toBe(false); }); it('reports applied status only after a real apply', () => { @@ -967,7 +987,7 @@ describe('StatisticAnalysisPanelComponent', () => { component.applyPreprocessing(); fixture.detectChanges(); - expect((component as any).userPreprocessingApplied).toBe(true); + expect(component.preprocessingState.userApplied).toBe(true); expect(component.preprocessingStatus).toBe('applied'); }); @@ -1105,7 +1125,7 @@ describe('StatisticAnalysisPanelComponent', () => { }); fixture.detectChanges(); - expect(component.processedSummary.featurewiseRows[0].data.mean).toBe(82); + expect(component.processedSummary.featurewiseRows[0].data).toMatchObject({ mean: 82 }); first.complete(); second.complete(); @@ -1143,7 +1163,7 @@ describe('StatisticAnalysisPanelComponent', () => { fixture.detectChanges(); expect(component.sourceSummary.featurewiseRows.length).toBe(1); - expect(component.sourceSummary.featurewiseRows[0].data.mean).toBe(82); + expect(component.sourceSummary.featurewiseRows[0].data).toMatchObject({ mean: 82 }); expect(component.sourceSummary.isLoading).toBe(false); stale.complete(); @@ -1817,7 +1837,7 @@ describe('StatisticAnalysisPanelComponent', () => { expect(component.ruleFor(age).action).toBe('median'); expect(component.ruleFor(sex).action).toBe('drop'); - expect((component as any).userPreprocessingApplied).toBe(false); + expect(component.preprocessingState.userApplied).toBe(false); // Internal status stays config-driven; the flag (and the emitted, // downgraded status) is what keeps the stepper from claiming Done. expect(component.preprocessingStatus).toBe('applied'); @@ -1857,7 +1877,7 @@ describe('StatisticAnalysisPanelComponent', () => { expect(component.prepVariableStateLabel('outlier', age)).toBe('Applied'); // Internal status stays config-driven (outlier config is applied); the // user-apply flag is what keeps the stepper from claiming Done. - expect((component as any).userPreprocessingApplied).toBe(false); + expect(component.preprocessingState.userApplied).toBe(false); expect(component.preprocessingStatus).toBe('applied'); expect(component.pendingChangeCount).toBe(0); }); @@ -1875,7 +1895,7 @@ describe('StatisticAnalysisPanelComponent', () => { (mockExpService.selectedVariables as any).set([aspiration]); fixture.detectChanges(); component.preprocessingStatus = 'applied'; - component.processedSummary = (component as any).buildSummaryFromResponse({ + component.processedSummary = component.statisticSummary.fromDescribeResponse({ result: { featurewise: [ { @@ -1890,7 +1910,7 @@ describe('StatisticAnalysisPanelComponent', () => { }, ], }, - }, 'processed'); + }, component.getSummary('processed'), component.preprocessingVariables); component.processedSummary.selectedStatisticKey = 'aspiration'; fixture.detectChanges(); @@ -2020,7 +2040,7 @@ describe('StatisticAnalysisPanelComponent', () => { }], }, })); - component.processedSummary = (component as any).buildSummaryFromResponse({ + component.processedSummary = component.statisticSummary.fromDescribeResponse({ result: { featurewise: [ { @@ -2030,7 +2050,7 @@ describe('StatisticAnalysisPanelComponent', () => { }, ], }, - }, 'processed'); + }, component.getSummary('processed'), component.preprocessingVariables); component.processedSummary.selectedStatisticKey = 'aspiration'; fixture.detectChanges(); @@ -2077,7 +2097,7 @@ describe('StatisticAnalysisPanelComponent', () => { // The pending edit the stored config does not carry yet. component.onMissingActionChange(age, 'mean'); - component.processedSummary = (component as any).buildSummaryFromResponse({ + component.processedSummary = component.statisticSummary.fromDescribeResponse({ result: { featurewise: [ { @@ -2087,7 +2107,7 @@ describe('StatisticAnalysisPanelComponent', () => { }, ], }, - }, 'processed'); + }, component.getSummary('processed'), component.preprocessingVariables); component.processedSummary.selectedStatisticKey = 'age'; fixture.detectChanges(); @@ -2400,11 +2420,10 @@ describe('StatisticAnalysisPanelComponent', () => { const draft = component.transformationDrafts[0]; expect(draft.rules.map((rule) => rule.value)).toEqual(['ACS', 'PCS']); expect(draft.rules.map((rule) => rule.filter)).toEqual([acs, pcs]); - expect(component.transformationDraftStatus(draft).label).toBe('Applied'); + expect(component.transformationDraftState.status(draft).label).toBe('Applied'); // Apply commits the rendered builders; a rule that loaded must survive untouched. - const modals: any = new QueryList(); - modals.reset([ + useRuleBuilders(component, [ { exportFilterLogic: () => ({ condition: 'AND', rules: [{ field: 'clinical_sdr', operator: 'in', value: ['1', '2'] }], valid: true }), filterError: () => null, @@ -2414,7 +2433,6 @@ describe('StatisticAnalysisPanelComponent', () => { filterError: () => null, }, ]); - component.transformationRuleModals = modals; component.commitTransformationDraft(draft); expect(draft.rules.map((rule) => (rule.filter as any)?.rules.length)).toEqual([1, 1]); @@ -2426,14 +2444,14 @@ describe('StatisticAnalysisPanelComponent', () => { const draft = component.transformationDrafts[0]; draft.code = 'elderly'; draft.rules = [{ value: 'old', filter }, { value: '', filter: null }]; - component.transformationRuleModals = new QueryList() as any; + useRuleBuilders(component, []); - expect(component.transformationDraftStatus(draft).label).toBe('Not applied'); + expect(component.transformationDraftState.status(draft).label).toBe('Not applied'); expect(component.transformationHasPendingChange).toBe(true); component.commitTransformationDraft(draft); expect(draft.rules.map((rule) => rule.value)).toEqual(['old']); - expect(component.transformationDraftStatus(draft).label).toBe('Applied'); + expect(component.transformationDraftState.status(draft).label).toBe('Applied'); expect(component.transformationHasPendingChange).toBe(false); draft.rules.push({ value: 'named-only', filter: null }); @@ -2446,15 +2464,15 @@ describe('StatisticAnalysisPanelComponent', () => { ensureCategoricalDrafts(1); const draft = component.transformationDrafts[0]; draft.code = 'elderly'; - component.countsDraftId = draft.id; - component.transformationStatistics = [{ code: 'elderly', rows: [{ value: 'old', count: 12 }] }]; - component.transformationStatisticsError = 'Failed to load category counts.'; + component.transformationStats.countsDraftId.set(draft.id); + component.transformationStats.blocks.set([{ code: 'elderly', rows: [{ value: 'old', count: 12 }] }]); + component.transformationStats.error.set('Failed to load category counts.'); component.removeTransformationRule(draft, 0); - expect(component.countsDraftId).toBeNull(); - expect(component.transformationStatistics).toEqual([]); - expect(component.transformationStatisticsError).toBe(''); + expect(component.transformationStats.countsDraftId()).toBeNull(); + expect(component.transformationStats.blocks()).toEqual([]); + expect(component.transformationStats.error()).toBe(''); }); it('hydrates every saved categorical creator into its own card', () => { @@ -2487,8 +2505,8 @@ describe('StatisticAnalysisPanelComponent', () => { // Both steps hydrate; the last one must not win. expect(component.transformationDrafts.map((draft) => draft.code)).toEqual(['group_a', 'group_b']); expect(component.transformationDrafts[1].defaultEnumeration).toBe('other'); - expect(component.transformationDraftStatus(component.transformationDrafts[0]).label).toBe('Applied'); - expect(component.transformationDraftStatus(component.transformationDrafts[1]).label).toBe('Applied'); + expect(component.transformationDraftState.status(component.transformationDrafts[0]).label).toBe('Applied'); + expect(component.transformationDraftState.status(component.transformationDrafts[1]).label).toBe('Applied'); // Re-persisting the same cards keeps them as an ordered array of two. mockExpService.upsertTransformation.mockClear(); @@ -2569,9 +2587,7 @@ describe('StatisticAnalysisPanelComponent', () => { const draft = component.transformationDrafts[0]; draft.code = 'mrs_good_outcome'; draft.rules = [{ value: 'good', filter }]; - component.transformationRuleModals = { - toArray: () => [{ exportFilterLogic: () => filter, filterError: () => null }], - } as any; + useRuleBuilders(component, [{ exportFilterLogic: () => filter, filterError: () => null }]); expect(component.sectionOpen().transformation).toBe(true); component.commitTransformationDraft(draft); @@ -2597,7 +2613,7 @@ describe('StatisticAnalysisPanelComponent', () => { // the stage around it any more. openStation('transformation'); ensureCategoricalDrafts(1); - component.transformationRuleModals = { toArray: () => [] } as any; + useRuleBuilders(component, []); component.commitTransformationDraft(component.transformationDrafts[0]); @@ -2618,7 +2634,7 @@ describe('StatisticAnalysisPanelComponent', () => { component.onTransformationChange(); expect(mockExpService.upsertTransformation).not.toHaveBeenCalled(); - expect(component.transformationDraftStatus(draft).label).toBe('Column in use'); + expect(component.transformationDraftState.status(draft).label).toBe('Column in use'); }); it('applies the first card when two derived columns share a name', () => { @@ -2630,12 +2646,10 @@ describe('StatisticAnalysisPanelComponent', () => { component.transformationDrafts[0].rules = [{ value: 'a', filter }]; component.transformationDrafts[1].code = 'group'; component.transformationDrafts[1].rules = [{ value: 'b', filter }]; - component.transformationRuleModals = { - toArray: () => [ + useRuleBuilders(component, [ { exportFilterLogic: () => filter, filterError: () => null }, { exportFilterLogic: () => filter, filterError: () => null }, - ], - } as any; + ]); fixture.detectChanges(); component.commitTransformationDraft(component.transformationDrafts[0]); @@ -2644,8 +2658,8 @@ describe('StatisticAnalysisPanelComponent', () => { // Apply no longer refuses. The second card owns the collision chip; only the first // occurrence of a shared code is persisted. - expect(component.transformationDraftStatus(component.transformationDrafts[0]).label).toBe('Applied'); - expect(component.transformationDraftStatus(component.transformationDrafts[1]).label).toBe('Duplicate name'); + expect(component.transformationDraftState.status(component.transformationDrafts[0]).label).toBe('Applied'); + expect(component.transformationDraftState.status(component.transformationDrafts[1]).label).toBe('Duplicate name'); expect(component.transformationStatusLabel).toBe('Not applied'); expect(navigate).not.toHaveBeenCalled(); expect(component.transformationDrafts[0].open).toBe(false); @@ -2666,12 +2680,10 @@ describe('StatisticAnalysisPanelComponent', () => { component.transformationDrafts[1].code = 'mrs_bad_outcome'; component.transformationDrafts[1].rules = [{ value: 'bad', filter: null }]; // One modal per rule, flattened in draft order, as the template emits them. - component.transformationRuleModals = { - toArray: () => [ + useRuleBuilders(component, [ { exportFilterLogic: () => filter, filterError: () => null }, { exportFilterLogic: () => secondFilter, filterError: () => null }, - ], - } as any; + ]); component.commitTransformationRuleFilters(); expect(component.transformationDrafts[0].rules[0].filter).toBe(filter as any); @@ -3139,7 +3151,7 @@ describe('StatisticAnalysisPanelComponent', () => { }); describe('Data Handling preview contracts', () => { - /** Stand-in for a category-rule QueryBuilder, as the template hands them over. */ + /** Stand-in for a category-rule builder (see useRuleBuilders). */ function ruleModal(filter: any, error: string | null = null) { return { exportFilterLogic: () => filter, @@ -3265,9 +3277,7 @@ describe('StatisticAnalysisPanelComponent', () => { unfiltered.code = 'other'; unfiltered.rules = [{ value: 'c', filter: null }]; nameless.defaultEnumeration = 'unknown'; - component.transformationRuleModals = { - toArray: () => [ruleModal(filter), ruleModal(filter), ruleModal(null)], - } as any; + useRuleBuilders(component, [ruleModal(filter), ruleModal(filter), ruleModal(null)]); mockExpService.loadDescriptiveOverview.mockReturnValue(of({ result: { featurewise: [ @@ -3284,7 +3294,7 @@ describe('StatisticAnalysisPanelComponent', () => { expect(transformation.querySelector('.row-error')).toBeNull(); expect(transformation.querySelector('.glass-feedback-warning')).toBeNull(); expect(stationButton('Transformation', '.station-action-apply').disabled).toBe(false); - expect(component.transformationDraftStatus(unfiltered).label).toBe('Not applied'); + expect(component.transformationDraftState.status(unfiltered).label).toBe('Not applied'); expect(component.transformationStatusLabel).toBe('Not applied'); mockExpService.loadDescriptiveOverview.mockClear(); @@ -3293,9 +3303,9 @@ describe('StatisticAnalysisPanelComponent', () => { // An incomplete card simply drops out of the describe; the complete ones // still get their counts, which each card charts for its own column. - expect(component.countsDraftId).toBe(first.id); + expect(component.transformationStats.countsDraftId()).toBe(first.id); expect(mockExpService.loadDescriptiveOverview).toHaveBeenCalledTimes(1); - expect(component.transformationStatistics).toEqual([ + expect(component.transformationStats.blocks()).toEqual([ { code: 'group_a', rows: [{ value: 'a', count: 3 }] }, { code: 'group_b', rows: [{ value: 'b', count: 4 }] }, ]); @@ -3345,7 +3355,7 @@ describe('StatisticAnalysisPanelComponent', () => { expect(mockExpService.buildTransformationDescribePreprocessing).toHaveBeenCalled(); expect(sourceCodes).toContain('age'); expect(sourceCodes).toContain('sex'); - expect(component.transformationStatistics).toEqual([{ + expect(component.transformationStats.blocks()).toEqual([{ code: 'kmeans_cluster', rows: [ { value: 'cluster_0', count: 12 }, @@ -3397,38 +3407,34 @@ describe('StatisticAnalysisPanelComponent', () => { expect((preprocessing as any[]).some((step) => step.name === 'kmeans_cluster_creator')).toBe(true); expect(sourceCodes).toContain('age'); expect(sourceCodes).toContain('bmi'); - expect(component.transformationStatistics).toEqual([{ + expect(component.transformationStats.blocks()).toEqual([{ code: 'km_fit', rows: [ { value: 'cluster_1', count: 9 }, { value: 'cluster_2', count: 4 }, ], }]); - expect(component.kmeansColumnView('km_fit')?.rules).toEqual({ + expect(component.transformationStats.kmeansColumnView('km_fit')?.rules).toEqual({ cluster_1: 'Nearest cluster', cluster_2: 'Nearest cluster', }); - expect(component.transformationStatisticsError).not.toContain('no clustering variables'); + expect(component.transformationStats.error()).not.toContain('no clustering variables'); }); it('keeps a rejected category open on Done without loading counts', () => { openStation('transformation'); // The card is on screen only once its type is chosen; its own footer owns Preview. - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); const draft = component.transformationDrafts[0]; draft.code = 'group_a'; draft.rules = [{ value: 'a', filter: null }]; - component.transformationRuleModals = { - toArray: () => [ruleModal(null, 'A filter value is required')], - } as any; + useRuleBuilders(component, [ruleModal(null, 'A filter value is required')]); fixture.detectChanges(); mockExpService.loadDescriptiveOverview.mockClear(); draft.openRuleIndex = 0; // detectChanges refreshes the @ViewChildren query; re-stub the rejecting builder. - component.transformationRuleModals = { - toArray: () => [ruleModal(null, 'A filter value is required')], - } as any; + useRuleBuilders(component, [ruleModal(null, 'A filter value is required')]); component.finishTransformationRule(draft, 0); fixture.detectChanges(); @@ -3436,7 +3442,7 @@ describe('StatisticAnalysisPanelComponent', () => { // and no describe runs for a half-typed condition. expect(draft.openRuleIndex).toBe(0); expect(draft.rules[0].filter).toBeNull(); - expect(component.countsDraftId).toBeNull(); + expect(component.transformationStats.countsDraftId()).toBeNull(); expect(mockExpService.loadDescriptiveOverview).not.toHaveBeenCalled(); }); @@ -3445,7 +3451,7 @@ describe('StatisticAnalysisPanelComponent', () => { ensureCategoricalDrafts(1); component.chooseTransformation('kmeans_cluster_creator'); component.transformationDrafts[0].code = 'pro'; - component.transformationRuleModals = { toArray: () => [] } as any; + useRuleBuilders(component, []); fixture.detectChanges(); component.previewTransformationCard(); @@ -3461,7 +3467,7 @@ describe('StatisticAnalysisPanelComponent', () => { openStation('transformation'); ensureCategoricalDrafts(1); component.transformationDrafts[0].code = 'pro'; - component.transformationRuleModals = { toArray: () => [] } as any; + useRuleBuilders(component, []); fixture.detectChanges(); mockExpService.loadDescriptiveOverview.mockClear(); @@ -3472,7 +3478,7 @@ describe('StatisticAnalysisPanelComponent', () => { // too. Before, it rendered dashes under "counts from a describe run" — a describe // that never happened. expect(mockExpService.loadDescriptiveOverview).not.toHaveBeenCalled(); - expect(component.transformationStatistics.map((block) => block.code)).toEqual(['pro']); + expect(component.transformationStats.blocks().map((block) => block.code)).toEqual(['pro']); const statistics = workflowSection('Transformation').querySelector('.transformation-statistics'); expect(statistics?.querySelector('.glass-feedback-warning')?.textContent?.trim()) .toBe('Each derived column needs a named category before counts can be loaded.'); @@ -3480,7 +3486,7 @@ describe('StatisticAnalysisPanelComponent', () => { }); it('turns category counts into a bar chart and skips a chart when every count is missing', () => { - expect(component.transformationCountChart({ + expect(component.transformationStats.countChart({ code: 'aaa', rows: [ { value: 'a', count: 88426 }, @@ -3492,7 +3498,7 @@ describe('StatisticAnalysisPanelComponent', () => { variableName: 'aaa', variableType: 'nominal', }); - expect(component.transformationCountChart({ + expect(component.transformationStats.countChart({ code: 'aaa', rows: [{ value: 'a', count: null }], })).toBeNull(); @@ -3512,9 +3518,9 @@ describe('StatisticAnalysisPanelComponent', () => { type: 'text/plain+error', }, })); - component.refreshTransformationStatistics(); - expect(component.transformationStatisticsError).toBe('Failed to load category counts.'); - expect(component.transformationStatisticsError).not.toContain('insufficient data'); + component.transformationStats.refresh(); + expect(component.transformationStats.error()).toBe('Failed to load category counts.'); + expect(component.transformationStats.error()).not.toContain('insufficient data'); mockExpService.loadDescriptiveOverview.mockReturnValue(of({ status: 'success', @@ -3522,8 +3528,8 @@ describe('StatisticAnalysisPanelComponent', () => { featurewise: [{ variable: 'cohort', dataset: 'all datasets', data: null }], }, })); - component.refreshTransformationStatistics(); - expect(component.transformationStatisticsError) + component.transformationStats.refresh(); + expect(component.transformationStats.error()) .toBe('No counts were returned (privacy threshold or insufficient data after preprocessing).'); }); @@ -3534,13 +3540,13 @@ describe('StatisticAnalysisPanelComponent', () => { unfiltered.code = 'group_a'; unfiltered.rules = [{ value: 'a', filter: null }]; nameless.code = 'group_b'; - component.transformationRuleModals = { toArray: () => [ruleModal(null), ruleModal(null)] } as any; + useRuleBuilders(component, [ruleModal(null), ruleModal(null)]); fixture.detectChanges(); component.finishTransformationRule(unfiltered, 0); // One line for the whole stage, covering each card's own shortcoming. - expect(component.transformationStatisticsError).toBe( + expect(component.transformationStats.error()).toBe( 'Each derived column needs a named category and a filter on each one before counts can be loaded.' ); }); @@ -3550,7 +3556,7 @@ describe('StatisticAnalysisPanelComponent', () => { ensureCategoricalDrafts(1); const draft = component.transformationDrafts[0]; draft.code = 'pro'; - component.transformationRuleModals = { toArray: () => [] } as any; + useRuleBuilders(component, []); component.onTransformationInput(); fixture.detectChanges(); @@ -3566,7 +3572,7 @@ describe('StatisticAnalysisPanelComponent', () => { // becomes the quiet Close. const filter = categoryFilter(); draft.rules = [{ value: 'old', filter }]; - component.transformationRuleModals = { toArray: () => [ruleModal(filter)] } as any; + useRuleBuilders(component, [ruleModal(filter)]); component.onTransformationInput(); fixture.detectChanges(); @@ -3584,9 +3590,7 @@ describe('StatisticAnalysisPanelComponent', () => { complete.rules = [{ value: 'a', filter }]; unfinished.code = 'group_b'; unfinished.rules = [{ value: 'b', filter: null }]; - component.transformationRuleModals = { - toArray: () => [ruleModal(filter), ruleModal(null)], - } as any; + useRuleBuilders(component, [ruleModal(filter), ruleModal(null)]); fixture.detectChanges(); component.commitTransformationDraft(complete); @@ -3615,7 +3619,7 @@ describe('StatisticAnalysisPanelComponent', () => { const draft = component.transformationDrafts[0]; draft.code = 'group_a'; draft.rules = [{ value: 'a', filter }]; - component.transformationRuleModals = { toArray: () => [ruleModal(filter)] } as any; + useRuleBuilders(component, [ruleModal(filter)]); mockExpService.loadDescriptiveOverview.mockReturnValue(of({ result: { featurewise: [ @@ -3629,7 +3633,7 @@ describe('StatisticAnalysisPanelComponent', () => { component.finishTransformationRule(draft, 0); fixture.detectChanges(); - expect(component.countsDraftId).toBe(draft.id); + expect(component.transformationStats.countsDraftId()).toBe(draft.id); expect(mockExpService.loadDescriptiveOverview).toHaveBeenCalledWith( ['group_a'], [ @@ -3641,7 +3645,7 @@ describe('StatisticAnalysisPanelComponent', () => { ['age'] ); expect(mockExpService.buildTransformationDescribePreprocessing).toHaveBeenCalled(); - expect(component.transformationStatistics).toEqual([ + expect(component.transformationStats.blocks()).toEqual([ { code: 'group_a', rows: [{ value: 'a', count: 7 }] }, ]); }); @@ -3672,7 +3676,7 @@ describe('StatisticAnalysisPanelComponent', () => { fixture.detectChanges(); const before = structuredClone(mockExpService.appliedTransformations()); - expect(component.columnNameError('pca-2', 'pca_pc1')).toBe('Name already used'); + expect(component.transformationDraftState.columnNameError('pca-2', 'pca_pc1')).toBe('Name already used'); expect(mockExpService.appliedTransformations()).toEqual(before); }); @@ -3683,14 +3687,14 @@ describe('StatisticAnalysisPanelComponent', () => { mockExpService.outputsInUse.mockImplementation((stepId: string, next: string[]) => stepId === id && !next.includes('grp') ? ['grp'] : [], ); - const draft = component.draftForStep(id)!; + const draft = component.transformationDraftState.draftFor(id)!; draft.code = 'grp_renamed'; mockExpService.upsertTransformation.mockClear(); - expect(component.columnNameError(id, 'grp_renamed')).toBe('Used by a later step: grp'); + expect(component.transformationDraftState.columnNameError(id, 'grp_renamed')).toBe('Used by a later step: grp'); component.onTransformationChange(); expect(mockExpService.upsertTransformation).not.toHaveBeenCalled(); - expect(component.transformationDraftStatus(draft).label).not.toBe('Duplicate name'); + expect(component.transformationDraftState.status(draft).label).not.toBe('Duplicate name'); }); it('disables remove when a step has dependents', () => { @@ -3715,7 +3719,7 @@ describe('StatisticAnalysisPanelComponent', () => { ); fixture.detectChanges(); - expect(component.canRemoveTransformation('pcaA')).toBe(false); + expect(component.transformationDraftState.canRemove('pcaA')).toBe(false); const removeBtn = workflowSection('Transformation').querySelector( '[data-step-id="pcaA"] .btn-remove-step-node', ) as HTMLButtonElement | null; diff --git a/src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.ts b/src/app/pages/experiment-studio/data-handling/data-handling.component.ts similarity index 54% rename from src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.ts rename to src/app/pages/experiment-studio/data-handling/data-handling.component.ts index 6e9afd316..b9c5b874f 100644 --- a/src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.component.ts +++ b/src/app/pages/experiment-studio/data-handling/data-handling.component.ts @@ -2,61 +2,52 @@ import { ChangeDetectionStrategy, ChangeDetectorRef, Component, - ElementRef, computed, effect, + forwardRef, inject, - QueryList, signal, untracked, - ViewChild, - ViewChildren, output, input, OnDestroy } from '@angular/core'; -import { NgTemplateOutlet } from '@angular/common'; import { FormsModule } from '@angular/forms'; import { EChartsOption } from 'echarts'; import { Subject, Subscription, takeUntil } from 'rxjs'; import { ExperimentStudioService, PreprocessingConfig } from '../../../services/experiment-studio.service'; -import { escapeHtml } from '../../../core/html.utils'; -import { ChartBuilderService } from '../visualisations/charts/chart-builder.service'; import { ChartRendererComponent } from '../visualisations/charts/charts-renderer/charts-renderer.component'; -import { HistogramComponent } from '../visualisations/histogram/histogram.component'; import { DATASET_VARIABLE_CODE, HISTOGRAM_SPLIT_NONE, - HistogramGroupSeries, HistogramSplitOption, buildHistogramSplitOptions, - buildStackedHistogramSeries, - extractHistogramItems, histogramCacheKey, } from '../visualisations/histogram/histogram-data'; -import { PdfExportService } from '../../../services/pdf-export.service'; -import { buildGroupedBarChart } from '../visualisations/charts/renderers/grouped-bar-chart'; -import { RuntimeEnvService } from '../../../services/runtime-env.service'; -import { - getFeaturewiseDescribeRows, - resolveDatasetDisplayLabel, -} from '../../../core/describe-result.utils'; import { FilterConfigModalComponent } from '../shared/filter-config-modal/filter-config-modal.component'; -import { StationActionBarComponent } from '../shared/station-action-bar/station-action-bar.component'; -import { StationCardComponent, StationStatus } from '../shared/station-card/station-card.component'; -import { KMeansClusterSourceComponent } from './kmeans-cluster-source/kmeans-cluster-source.component'; -import { PcaColumnSourceComponent } from './pca-column-source/pca-column-source.component'; -import { - kmeansCreatorClusterChoices, - kmeansCreatorVariables, -} from '../../../core/kmeans-cluster-source.utils'; -import { BackendFilter, BackendRule } from '../../../models/filters.model'; -import { CsvExportService } from '../../../services/csv-export.service'; +import { StationStatus } from '../shared/station-card/station-card.component'; +import { DATA_HANDLING } from './data-handling.context'; +import { TransformationStationComponent } from './transformation-station/transformation-station.component'; +import { PreprocessingStationComponent } from './preprocessing-station/preprocessing-station.component'; +import { CohortFilterStationComponent } from './cohort-filter-station/cohort-filter-station.component'; +import { SourceDataStationComponent } from './source-data-station/source-data-station.component'; +import { SectionAnchorDirective, SectionAnchors } from './section-anchors'; +import { RuleBuilder, RuleBuilderRegistry } from './transformation/rule-builder-registry'; +import { datasetDisplayLabel, emptySummaryView, selectionKey, statisticBlockKey, summaryKindLabel } from './utils/summary-view.utils'; +import { CLOSED_SECTIONS, StageStep, isStepShown, sectionLayout } from './utils/section-layout.utils'; +import { cloneMissingRules, defaultMissingRule, emptyMissingRule, groupPreprocessingVariables, serializeMissingRule } from './utils/preprocessing-state.utils'; +import { PreprocessingStateService } from './preprocessing-station/preprocessing-state.service'; +import { StatisticSummaryService } from './statistic-summary.service'; +import { SummaryExportService } from './summary-export.service'; +import { SummaryHistogramService } from './summary-histogram.service'; +import { enumLabelMap, isCategoricalVariable, pooledTotalRows, variableLabel } from './utils/statistic-summary.utils'; +import { PipelineTerminalComponent } from './pipeline-terminal/pipeline-terminal.component'; +import { BackendFilter } from '../../../models/filters.model'; +import { kmeansCreatorClusterChoices, kmeansCreatorVariables } from '../../../core/kmeans-cluster-source.utils'; import { ExperimentStudioNavigationService } from '../../../services/experiment-studio-navigation.service'; -import { countFilterRules, formatFilterExpression } from '../../../core/filter-display.utils'; +import { countFilterRules } from '../../../core/filter-display.utils'; import { ExperimentStudioGuideStateService } from '../guide/experiment-studio-guide-state.service'; import { getAnalysisGuideLayout } from '../guide/experiment-studio-analysis-guide.util'; -import { AlgorithmNames } from '../../../core/constants/algorithm.constants'; import { pluralize } from '../../../core/result-label.utils'; import { filterOperatorPhrase } from '../../../core/filter-logic.utils'; import { @@ -72,270 +63,186 @@ import { OutlierTail, outlierStrategyLabel, outlierTailLabel, - serializeOutlierRule, serializeOutlierRules, validateOutlierRule, } from '../../../core/outlier-rules'; -import { collectVariableCodes } from '../../../core/variable-availability.utils'; import { CreatorName, TransformationStep, - newStepId, stepOutputNames as stepOutputNamesOf, } from '../../../core/transformation-steps.utils'; -type TabKey = 'Statistics' | 'Charts' | 'Histogram'; -type SummaryKind = 'source' | 'raw' | 'processed'; -type SectionKey = 'source' | 'raw' | 'setup' | 'filters' | 'processed' | 'transformation'; -type PipelineStep = 'source' | 'filters' | 'setup' | 'transformation'; -type PrepStep = 'missing' | 'outlier' | 'longitudinal'; -type DistributionSubTab = 'Numeric' | 'Nominal'; -type StatisticVariableType = 'numeric' | 'nominal'; -type PreprocessingStatus = 'none' | 'pending' | 'applied'; -type PrepKind = 'missing' | 'outlier' | 'longitudinal'; - -type SummaryAnchorKey = 'charts' | 'details' | 'export'; - -/** - * Where each summary surface wears a `data-guide` anchor. They are not uniform: the Raw - * surface owns the overlay-tab and export anchors, the Processed surface only its details - * panel, and step 0 is a read-only duplicate of the Raw surface, so it wears none — the - * same anchor on a hidden node would hand the tour or a QA selector the wrong surface. - */ -/** Backend step docs arrive as prose whose bullets are '-', '*' or '\u2022'; all of them - become real list items, so the block reads like the algorithm documentation. */ -const DOCUMENTATION_BULLET = /^[-*\u2022\u00b7]\s+/; - -const SUMMARY_GUIDE_ANCHORS: Record> = { - source: { charts: null, details: null, export: null }, - raw: { charts: 'analysis-charts', details: 'analysis-raw-details', export: 'analysis-export' }, - processed: { charts: null, details: 'analysis-processed-details', export: null }, -}; - -type MissingAction = 'no_action' | 'drop' | 'mean' | 'median' | 'constant'; -type LongitudinalStrategy = 'first' | 'second' | 'diff'; - -/** - * The record-level rule behind one category. exaflow takes either a group of rules or a - * single condition, and saved experiments hold both shapes - so the draft keeps whichever - * it was given instead of pretending a bare condition cannot exist. - */ -type CategoryFilter = BackendFilter | BackendRule; - -interface TransformationRule { - value: string; - filter: CategoryFilter | null; -} - -/** One "Create new categorical column" card. The stage holds an ordered list. */ -interface TransformationColumnDraft { - /** Matches the store step id so cards() and drafts stay aligned. */ - id: string; - code: string; - defaultEnumeration: string; - rules: TransformationRule[]; - open: boolean; - /** Rule index whose filter editor is expanded; at most one open per card. */ - openRuleIndex?: number | null; - /** Picked from the type chooser (or restored from the store). */ - chosen?: boolean; -} - -const CREATOR_CHOOSER_COPY: Record = { - categorical_column_creator: { - icon: 'fas fa-tags', - title: 'Nominal column from rules', - subtitle: 'Name categories and pick each one\'s records with a filter — e.g. MMSE below 24 → ‘impaired’', - }, - kmeans_cluster_creator: { - icon: 'fas fa-diagram-project', - title: 'K-means cluster column', - subtitle: 'Group records by similarity and use the cluster as a nominal variable', - }, - pca_column_creator: { - icon: 'fas fa-chart-line', - title: 'PCA component columns', - subtitle: 'Project numerical variables onto principal components', - }, -}; - -/** A summary cell as a number: tolerates grouping commas and a trailing %; null when not numeric. */ -function parseNumber(value: unknown): number | null { - const text = String(value ?? '').replace(/[,%]/g, '').trim(); - const number = Number(text); - return text !== '' && Number.isFinite(number) ? number : null; -} - -function emptyTransformationDraft(id = newStepId()): TransformationColumnDraft { - return { id, code: '', defaultEnumeration: '', rules: [], open: true, openRuleIndex: null }; -} - -/** One sub-step of a pipeline stage, rendered on the pipeline-overview rail. */ -interface PipelineSubNode { - // Preprocessing stations use fixed ids; transformation sub-nodes carry a - // `transformation:` id so the rail can list one card per derived column. - id: string; - icon: string; - title: string; - subtitle: string; - statusLabel: string; - statusTone: 'applied' | 'default' | 'pending'; - /** Optional consequence line (e.g. rows the default drop-NaN would remove). */ - impact?: string; -} - -type MetricKey = - | 'num_dtps' | 'num_na' | 'num_total' - | 'mean' | 'std' | 'min' | 'q1' | 'q2' | 'q3' | 'max'; - -interface SummaryView { - activeTab: TabKey; - data: PivotBlock[]; - featurewiseRows: any[]; - isLoading: boolean; - showDistributions: boolean; - distributionSubTab: DistributionSubTab; - nonNominalVariables: VariableRow[]; - nominalVariables: VariableRow[]; - chartsForBoxPlot: EChartsOption[][]; - chartsForNominal: EChartsOption[][]; - histogramDataByVariable: Record; - histogramLoadingByVariable: Record; - histogramErrorByVariable: Record; - histogramSubscriptionsByVariable: Record; - histogramRequestIdByVariable: Record; - histogramSplitByVariable: Record; - activeBoxPlotIndex: number; - activeNominalIndex: number; - selectedStatisticKey: string | null; -} - -interface PivotBlock { - code?: string; - name: string; - columns: string[]; - rows: Array<{ metric: string; values: Record }>; -} - -interface VariableRow { - code: string; - name?: string; - label?: string; - type?: string; - enumerations?: Array<{ code?: string; label?: string; name?: string }>; -} - -interface EnumOption { - code: string; - label: string; -} - -interface HistogramPreviewData { - bins: string[]; - counts: Array; - variableName: string; - variableType?: string; - groups?: HistogramGroupSeries[]; - splitLabel?: string; -} - -interface PreprocessingRule { - variableCode: string; - action: MissingAction; - value: string; - enabled: boolean; -} - -interface PreprocessingGroup { - key: 'pending' | 'applied' | 'default' | 'not-applied'; - title: string; - variables: VariableRow[]; -} - -interface OutlierPreviewRow { - variable: string; - dataset: string; - strategy: string; - tail: string; - fold: string; - lowerBound: string; - upperBound: string; - lowerOutliers: string; - upperOutliers: string; - totalOutliers: string; - outlierPercentage: string; -} - -export interface DescriptiveProgressState { - pendingChangeCount: number; - preprocessingStatus: PreprocessingStatus; - transformationStatusLabel: string; -} +import { + TabKey, + SummaryKind, + SectionKey, + PipelineStep, + PrepStep, + StatisticVariableType, + PreprocessingStatus, + PrepKind, + SummaryAnchorKey, + SUMMARY_GUIDE_ANCHORS, + MissingAction, + LongitudinalStrategy, + CategoryFilter, + TransformationRule, + TransformationColumnDraft, + PipelineSubNode, + SummaryView, + PivotBlock, + VariableRow, + EnumOption, + HistogramPreviewData, + HowToGuide, + PreprocessingRule, + PreprocessingGroup, + OutlierPreviewRow, + DescriptiveProgressState, +} from './data-handling.models'; +import { buildPreprocessingDocumentationHtml } from './utils/preprocessing-documentation.utils'; +import { OutlierBoundsView, outlierBoundsView, outlierDistributionNumbers } from './utils/outlier-bounds.utils'; +import { HistogramPreviewVariable } from './utils/histogram-preview.utils'; +import { CREATOR_CHOOSER_COPY, firstFilterCondition, transformationConfigFor } from './utils/transformation-draft.utils'; +import { TransformationDraftsService } from './transformation-station/transformation-drafts.service'; +import { TransformationStatisticsService } from './transformation-station/transformation-statistics.service'; +import { + CategoryShare, + Coverage, + DatasetCoverage, + DistributionPoint, + PooledCoverage, + barWidth, + blockCoverage, + blockDistribution, + categoryShares, + coverageByDataset, + coveragePercent, + coverageSeverity, + coverageTitle, + metricGroupLabel, + missingShareNote, + pooledCoverage, +} from './utils/summary-coverage.utils'; +import { + displayNumber, + formatInteger, + formatOutlierBound, + formatOutlierPercentage, +} from './utils/statistic-format.utils'; @Component({ - selector: 'app-statistic-analysis-panel', - imports: [ChartRendererComponent, HistogramComponent, FormsModule, NgTemplateOutlet, FilterConfigModalComponent, StationActionBarComponent, StationCardComponent, KMeansClusterSourceComponent, PcaColumnSourceComponent], - templateUrl: './statistic-analysis-panel.component.html', + selector: 'app-data-handling', + imports: [ChartRendererComponent, FormsModule, PipelineTerminalComponent, SectionAnchorDirective, SourceDataStationComponent, CohortFilterStationComponent, PreprocessingStationComponent, TransformationStationComponent], + // Section components read the pipeline's state through this token (see data-handling.context.ts). + providers: [ + { provide: DATA_HANDLING, useExisting: forwardRef(() => DataHandlingComponent) }, + RuleBuilderRegistry, + SectionAnchors, + TransformationDraftsService, + TransformationStatisticsService, + PreprocessingStateService, + StatisticSummaryService, + SummaryHistogramService, + SummaryExportService, + ], + templateUrl: './data-handling.component.html', // Order is the cascade: shell + pipeline canvas, then preprocessing stations, // then shared controls / result surfaces + responsive overrides. Concatenated in this order. styleUrls: [ - './statistic-analysis-panel.component.css', - './statistic-analysis-panel.preprocessing.css', - './statistic-analysis-panel.results.css', + './data-handling.component.css', + './data-handling.preprocessing.css', + './data-handling.results.css', ], host: { '(document:click)': 'closeBatchMenuOnOutsideClick($event)', }, changeDetection: ChangeDetectionStrategy.OnPush, }) -export class StatisticAnalysisPanelComponent implements OnDestroy { +export class DataHandlingComponent implements OnDestroy { readonly processedDataInput = input([], { alias: 'processedData' }); processedData: PivotBlock[] = []; readonly variables = input([]); readonly filters = input([]); readonly progressStateChange = output(); - @ViewChild('sourceSection') - sourceSection?: ElementRef; - @ViewChild('rawSection') - rawSection?: ElementRef; - @ViewChild('setupSection') - setupSection?: ElementRef; - @ViewChild('filtersSection') - filtersSection?: ElementRef; - @ViewChild('processedSection') - processedSection?: ElementRef; - @ViewChild('transformationSection') - transformationSection?: ElementRef; - @ViewChildren('ruleModal') - transformationRuleModals?: QueryList; + /** Scroll targets of goToSection, registered by the elements themselves. */ + private readonly sectionAnchors = inject(SectionAnchors); + /** Each category's live filter builder, by rule (see RuleBuilderRegistry). */ + readonly ruleBuilders = inject(RuleBuilderRegistry); + readonly transformationDraftState = inject(TransformationDraftsService); + readonly transformationStats = inject(TransformationStatisticsService); + readonly preprocessingState = inject(PreprocessingStateService); + readonly statisticSummary = inject(StatisticSummaryService); + private readonly summaryHistogram = inject(SummaryHistogramService); + private readonly summaryExport = inject(SummaryExportService); expStudioService = inject(ExperimentStudioService); private readonly destroyed = new Subject(); private studioNavigation = inject(ExperimentStudioNavigationService); - private chartBuilder = inject(ChartBuilderService); - private pdfExportService = inject(PdfExportService); - private csvExportService = inject(CsvExportService); private cdr = inject(ChangeDetectorRef); - private runtimeEnvService = inject(RuntimeEnvService); - readonly mipVersion = this.runtimeEnvService.mipVersion; - rawSummary = this.createEmptySummary(true); - processedSummary = this.createEmptySummary(false); + get rawSummary(): SummaryView { + return this.statisticSummary.raw; + } + set rawSummary(view: SummaryView) { + this.statisticSummary.raw = view; + } + get processedSummary(): SummaryView { + return this.statisticSummary.processed; + } + set processedSummary(view: SummaryView) { + this.statisticSummary.processed = view; + } /** Step 0 snapshot: the same `SummaryView` the Raw and Processed surfaces use. */ - sourceSummary = this.createEmptySummary(true); - pendingPreprocessingRules: Record = {}; - appliedPreprocessingRules: Record = {}; + get sourceSummary(): SummaryView { + return this.statisticSummary.source; + } + set sourceSummary(view: SummaryView) { + this.statisticSummary.source = view; + } + // The rule maps live in PreprocessingStateService; these keep the panel's names. + get pendingPreprocessingRules(): Record { + return this.preprocessingState.pendingMissing; + } + set pendingPreprocessingRules(value: Record) { + this.preprocessingState.pendingMissing = value; + } + get appliedPreprocessingRules(): Record { + return this.preprocessingState.appliedMissing; + } + set appliedPreprocessingRules(value: Record) { + this.preprocessingState.appliedMissing = value; + } + get pendingLongitudinalStrategies(): Record { + return this.preprocessingState.pendingLongitudinal; + } + set pendingLongitudinalStrategies(value: Record) { + this.preprocessingState.pendingLongitudinal = value; + } + get appliedLongitudinalStrategies(): Record { + return this.preprocessingState.appliedLongitudinal; + } + set appliedLongitudinalStrategies(value: Record) { + this.preprocessingState.appliedLongitudinal = value; + } + get pendingOutlierRules(): Record { + return this.preprocessingState.pendingOutlier; + } + set pendingOutlierRules(value: Record) { + this.preprocessingState.pendingOutlier = value; + } + get appliedOutlierRules(): Record { + return this.preprocessingState.appliedOutlier; + } + set appliedOutlierRules(value: Record) { + this.preprocessingState.appliedOutlier = value; + } appliedLongitudinalEnabled = false; longitudinalVisit1 = ''; longitudinalVisit2 = ''; appliedLongitudinalVisit1 = ''; appliedLongitudinalVisit2 = ''; - pendingLongitudinalStrategies: Record = {}; - appliedLongitudinalStrategies: Record = {}; - pendingOutlierRules: Record = {}; - appliedOutlierRules: Record = {}; preprocessingStatus: PreprocessingStatus = 'none'; - private userPreprocessingApplied = false; isApplyingPreprocessing = false; preprocessingValidationErrors: Record = {}; outlierValidationErrors: Record = {}; @@ -360,16 +267,11 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { raw: '', processed: '', }; - isExporting = false; + get isExporting(): boolean { + return this.summaryExport.exporting(); + } isLoading = true; - readonly sectionOpen = signal>({ - source: false, - raw: false, - setup: false, - filters: false, - processed: false, - transformation: false, - }); + readonly sectionOpen = signal>({ ...CLOSED_SECTIONS }); readonly summaryExpanded = signal<{ filters: boolean; setup: boolean }>({ filters: false, setup: false, @@ -381,14 +283,14 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { transformation: true, }; - readonly addedSteps = signal>({ + readonly addedSteps = signal>({ filters: false, setup: false, transformation: false, }); readonly showUnappliedChangesWarning = signal(false); - isStepAdded(step: 'filters' | 'setup' | 'transformation'): boolean { + isStepAdded(step: StageStep): boolean { if (this.addedSteps()[step]) return true; if (step === 'filters') return this.filterRuleCount() > 0; if (step === 'transformation') { @@ -408,14 +310,14 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } /** Opening a stage is a read: it shows the editor, it never writes the request. */ - addStep(step: 'filters' | 'setup' | 'transformation'): void { + addStep(step: StageStep): void { this.addedSteps.update((s) => ({ ...s, [step]: true })); if (step === 'setup') this.ensureDefaultRulesForCurrentSelection(); this.goToSection(step); this.cdr.markForCheck(); } - removeStep(step: 'filters' | 'setup' | 'transformation'): void { + removeStep(step: StageStep): void { this.addedSteps.update((s) => ({ ...s, [step]: false })); if (step === 'filters') { this.rawPreviewFilter = undefined; @@ -429,11 +331,10 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.userEnabledOutliers.set(false); this.persistAppliedDescriptivePreprocessing(null); this.preprocessingStatus = 'none'; - this.userPreprocessingApplied = false; + this.preprocessingState.userApplied = false; this.processedDescribeSeq++; this.clearProcessedPreviewSubscription(); - this.clearHistogramSubscriptions(this.processedSummary); - this.processedSummary = this.createEmptySummary(false); + this.processedSummary = emptySummaryView(false); this.processedSummaryKey = ''; this.sectionOpen.update((open) => ({ ...open, setup: false, processed: false })); } else if (step === 'transformation') { @@ -454,16 +355,16 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return; } this.removeStep('filters'); - this.requestSectionScroll('filters'); + this.sectionAnchors.requestScroll('filters'); } /** Folds the stage's editor and preview; nothing is removed or discarded. */ - collapseStep(step: 'filters' | 'setup' | 'transformation'): void { + collapseStep(step: StageStep): void { if (step === 'filters') return this.foldFilterStation(); if (step === 'setup') return this.foldPreprocessingStation(); this.sectionOpen.update((open) => ({ ...open, transformation: false })); this.cdr.markForCheck(); - this.requestSectionScroll('transformation'); + this.sectionAnchors.requestScroll('transformation'); } /** A click anywhere on a step's header toggles it, except on the header's own buttons @@ -483,19 +384,15 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { /** A step outside the pipeline is never open, whatever its section flag says. */ isStepOpen(step: PipelineStep): boolean { - const open = this.sectionOpen(); - if (step === 'source') return open.source; - if (!this.isStepAdded(step)) return false; - if (step === 'filters') return open.filters || open.raw; - if (step === 'setup') return open.setup || open.processed; - return open.transformation; + if (step !== 'source' && !this.isStepAdded(step)) return false; + return isStepShown(step, this.sectionOpen()); } /** Apply saved the rules; fold the editor. Preview data is what opens the tables. */ private foldFilterStation(): void { this.sectionOpen.update((open) => ({ ...open, filters: false, raw: false })); this.cdr.markForCheck(); - this.requestSectionScroll('filters'); + this.sectionAnchors.requestScroll('filters'); } /** @@ -710,13 +607,14 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { }); continue; } + const available = this.transformationDraftState.isCreatorAvailable(step.name); nodes.push({ id: `transformation:${step.id}`, - icon: CREATOR_CHOOSER_COPY[step.name]?.icon ?? 'fas fa-square-plus', - title: `${CREATOR_CHOOSER_COPY[step.name]?.title ?? step.name}: ${outputs.join(', ')}`, - subtitle: this.isCreatorAvailable(step.name) ? 'Configured' : 'Not available on this deployment', - statusLabel: this.isCreatorAvailable(step.name) ? 'Applied' : 'Unavailable', - statusTone: this.isCreatorAvailable(step.name) ? 'applied' : 'pending', + icon: CREATOR_CHOOSER_COPY[step.name].icon, + title: `${CREATOR_CHOOSER_COPY[step.name].title}: ${outputs.join(', ')}`, + subtitle: available ? 'Configured' : 'Not available on this deployment', + statusLabel: available ? 'Applied' : 'Unavailable', + statusTone: available ? 'applied' : 'pending', }); } return nodes; @@ -751,40 +649,9 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.addStep('transformation'); } - isDraftShown(draft: TransformationColumnDraft): boolean { - return !!draft.chosen || this.draftHasWork(draft); - } - /** Ordered cards from the store — one station per transformation step. */ cards(): TransformationStep[] { - return this.expStudioService.appliedTransformations(); - } - - /** Template helper: output codes a step writes (`new_column_names`). */ - stepOutputNames(step: TransformationStep): string[] { - return stepOutputNamesOf(step); - } - - /** Creator types the deployment exposes in `/services/specifications/preprocessing`. */ - chooserCreators(): CreatorName[] { - return this.expStudioService.availableCreatorNames(); - } - - creatorChooserCopy(kind: CreatorName): { icon: string; title: string; subtitle: string } { - return CREATOR_CHOOSER_COPY[kind]; - } - - /** - * True when the editor should render for this creator. Empty specs before the catalog - * loads are not "unavailable" — only loaded && missing means the deployment placeholder. - */ - isCreatorAvailable(name: CreatorName): boolean { - if (!this.expStudioService.catalogLoaded()) return true; - return this.expStudioService.hasPreprocessingSpec(name); - } - - get kmeansCardShown(): boolean { - return this.cards().some((step) => step.name === 'kmeans_cluster_creator'); + return this.transformationDraftState.cards(); } /** Nothing picked yet: the chooser is the whole stage, so it asks instead of offering. */ @@ -792,110 +659,17 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return this.cards().length === 0; } - /** - * Type chooser row: always appends a new store step (fresh id). Categorical cards also get a - * matching local draft for the rule editor. - */ - chooseTransformation(kind: CreatorName | 'categorical' | 'kmeans' | 'pca'): void { - const name = this.resolveCreatorName(kind); - const id = newStepId(); - this.expStudioService.upsertTransformation({ id, name, parameters: {} }); - if (name === 'categorical_column_creator') { - this.transformationDrafts = [ - ...this.transformationDrafts.filter((draft) => this.isDraftShown(draft)), - { ...emptyTransformationDraft(id), chosen: true, open: true }, - ]; - } - this.cardOpenState.update((state) => ({ ...state, [id]: true })); + /** Type chooser row: appends a new store step (a nominal one also gets its rule-editor draft). */ + chooseTransformation(name: CreatorName): void { + this.transformationDraftState.choose(name); this.cdr.markForCheck(); } - private resolveCreatorName(kind: CreatorName | 'categorical' | 'kmeans' | 'pca'): CreatorName { - if (kind === 'categorical') return 'categorical_column_creator'; - if (kind === 'kmeans') return 'kmeans_cluster_creator'; - if (kind === 'pca') return 'pca_column_creator'; - return kind; - } - - draftForStep(stepId: string): TransformationColumnDraft | undefined { - return this.transformationDrafts.find((draft) => draft.id === stepId); - } - - columnNameError(stepId: string, name: string): string | null { - const trimmed = name.trim(); - if (!trimmed) return null; - const cdeCodes = new Set(collectVariableCodes(this.expStudioService.selectedDataModel())); - if (cdeCodes.has(trimmed)) return 'Name already used'; - for (const step of this.cards()) { - if (step.id === stepId) continue; - // Ignore empty/incomplete placeholders — only real outputs collide. - if (stepOutputNamesOf(step).includes(trimmed)) return 'Name already used'; - } - // Among local categorical drafts, the first occurrence of a code wins (same as the - // old payload dedupe); later cards show the error and never overwrite the store. - for (const draft of this.transformationDrafts) { - if (draft.id === stepId) break; - if (draft.code.trim() === trimmed) return 'Name already used'; - } - // A nominal card writes one column: renaming it must not orphan a later step that reads it. - if (this.draftForStep(stepId)) { - const inUse = this.expStudioService.outputsInUse(stepId, [trimmed]); - if (inUse.length) return `Used by a later step: ${inUse.join(', ')}`; - } - return null; - } - - /** Stable callback for K-means / PCA cards (`[columnNameError]` input). */ - readonly kmeansColumnNameError = (stepId: string, name: string): string | null => - this.columnNameError(stepId, name); - - canRemoveTransformation(stepId: string): boolean { - return this.expStudioService.transformationDependents(stepId).length === 0; - } - - removeTransformationTooltip(stepId: string): string { - const dependents = this.expStudioService.transformationDependents(stepId); - if (!dependents.length) return 'Remove column'; - const names = dependents.flatMap((step) => stepOutputNamesOf(step)); - return names.length ? `Used by ${names.join(', ')}` : 'Used by a later step'; - } - - isCardOpen(stepId: string): boolean { - return this.cardOpenState()[stepId] ?? true; - } - - toggleCardOpen(stepId: string, open?: boolean): void { - this.cardOpenState.update((state) => ({ - ...state, - [stepId]: open ?? !this.isCardOpen(stepId), - })); + toggleCardOpen(stepId: string, open = !this.transformationDraftState.isCardOpen(stepId)): void { + this.transformationDraftState.setCardOpen(stepId, open); this.cdr.markForCheck(); } - stepStatus(step: TransformationStep): { label: string; tone: StationStatus } { - if (!this.isCreatorAvailable(step.name)) { - return { label: 'Unavailable', tone: 'pending' }; - } - if (step.name === 'categorical_column_creator') { - const draft = this.draftForStep(step.id); - return draft ? this.transformationDraftStatus(draft) : { label: 'Not set', tone: 'default' }; - } - if (stepOutputNamesOf(step).length) return { label: 'Applied', tone: 'applied' }; - return { label: 'Not set', tone: 'default' }; - } - - stepTitle(step: TransformationStep): string { - const copy = CREATOR_CHOOSER_COPY[step.name]; - const outputs = stepOutputNamesOf(step); - if (step.name === 'categorical_column_creator') { - const draft = this.draftForStep(step.id); - const code = draft?.code.trim() || outputs[0] || ''; - return code ? `Nominal column · ${code}` : 'Nominal column'; - } - if (outputs.length) return `${copy.title} · ${outputs[0]}`; - return copy.title; - } - private missingValuesSubNode(config: Record): PipelineSubNode { const handler = config['missing_values_handler'] as | { strategies?: Record } @@ -1007,7 +781,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } /** "How to use" strip copy per expanded stage; K-means is projected into its card. */ - readonly howTo: Record<'source' | 'filters' | 'setup' | 'transformation' | 'kmeans' | 'pca', { steps: string[]; note: string }> = { + readonly howTo: Record<'source' | 'filters' | 'setup' | 'transformation' | 'kmeans' | 'pca', HowToGuide> = { source: { steps: ['Pick a variable on the left', 'Switch between Table, Charts and Histogram', 'Note gaps or odd ranges — fix them in the steps below'], note: 'Read only — this preview never changes the data or the run.', @@ -1043,16 +817,14 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { /** The card whose category counts replace its editor ('kmeans' = the cluster card). */ previewDraftId: string | null = null; - /** The categorical card whose Done loaded the current counts; it shows their loading/error state. */ - countsDraftId: string | null = null; /** Ordered cards; the stage always keeps at least one (possibly empty) draft. */ - transformationDrafts: TransformationColumnDraft[] = []; - /** Collapsed/expanded state per store step id (categorical drafts also keep `open`). */ - readonly cardOpenState = signal>({}); - transformationStatistics: Array<{ code: string; rows: Array<{ value: string; count: number | null }> }> = []; - isTransformationStatsLoading = false; - transformationStatisticsError = ''; - private transformationStatsRequestId = 0; + /** The nominal-column cards (state and rules in TransformationDraftsService). */ + get transformationDrafts(): TransformationColumnDraft[] { + return this.transformationDraftState.drafts; + } + set transformationDrafts(drafts: TransformationColumnDraft[]) { + this.transformationDraftState.drafts = drafts; + } readonly longitudinalStrategies: Array<{ value: LongitudinalStrategy; label: string }> = [ { value: 'diff', label: 'Diff (Visit 2 - Visit 1)' }, @@ -1062,32 +834,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { readonly outlierStrategies = OUTLIER_STRATEGIES; readonly outlierTails = OUTLIER_TAILS; - readonly metricOrder: Array<{ key: string; label: string }> = [ - { key: 'num_datapoints', label: 'Datapoints' }, - { key: 'num_missing', label: 'Missing' }, - { key: 'num_total', label: 'Total' }, - { key: 'mean', label: 'Mean' }, - { key: 'std', label: 'Standard Deviation' }, - { key: 'min', label: 'Min' }, - { key: 'q1', label: 'Q1' }, - { key: 'q2', label: 'Median' }, - { key: 'q3', label: 'Q3' }, - { key: 'max', label: 'Max' }, - ]; - - readonly metricLabel: Record = { - num_dtps: 'Datapoints', - num_na: 'Missing', - num_total: 'Total', - mean: 'Mean', - std: 'Standard Deviation', - min: 'Min', - q1: 'Q1', - q2: 'Median', - q3: 'Q3', - max: 'Max', - }; - private selectionKey = ''; /** Cohort half of `selectionKey`, so a preprocessing write can be ignored on its own. */ private cohortSelectionKey = ''; @@ -1099,7 +845,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { private appliedConfigWrite = ''; /** Card preview describes that card's pending rules over the applied rest. */ private processedPreviewScope: PrepStep | 'all' = 'all'; - private rawSummaryKey = ''; + private get rawSummaryKey(): string { + return this.statisticSummary.loadedKey.raw; + } + private set rawSummaryKey(key: string) { + this.statisticSummary.loadedKey.raw = key; + } /** * The cohort rules the Raw surface was last fetched with when they are unsaved * editor state. `undefined` means the surface follows the stored `filterLogic()`, @@ -1110,19 +861,50 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { private rawPreviewFilter: BackendFilter | null | undefined = undefined; /** Supersedes an in-flight raw describe so a late response cannot win. */ private rawDescribeSeq = 0; - private processedSummaryKey = ''; - private sourceSummaryKey = ''; + private get processedSummaryKey(): string { + return this.statisticSummary.loadedKey.processed; + } + private set processedSummaryKey(key: string) { + this.statisticSummary.loadedKey.processed = key; + } + private get sourceSummaryKey(): string { + return this.statisticSummary.loadedKey.source; + } + private set sourceSummaryKey(key: string) { + this.statisticSummary.loadedKey.source = key; + } private processedPreviewSubscription: Subscription | null = null; private sourcePreviewSubscription: Subscription | null = null; /** Supersedes in-flight processed/source describes so a late response cannot win. */ private processedDescribeSeq = 0; private sourceDescribeSeq = 0; - private scrollRequestId = 0; private continueAfterApply = false; - private readonly scrollSectionRequest = signal<{ section: SectionKey; requestId: number } | null>(null); private readonly guideState = inject(ExperimentStudioGuideStateService); constructor() { + this.transformationStats.ports = { + hasInvalidPendingRules: () => Object.keys(this.validatePendingRules()).length > 0, + pendingPreprocessingConfig: () => this.buildPreprocessingConfig(this.pendingPreprocessingRules), + variableLabel: (code) => this.variableLabelForCode(code), + }; + this.summaryHistogram.ports = { + selectedBlock: (kind) => this.selectedStatisticBlock(kind), + target: (kind, block) => { + const code = this.variableCodeForBlock(block); + if (!code) return null; + return { + code, + split: this.histogramSplitValue(kind, block), + variable: () => this.histogramVariable(kind, block), + splitLabel: (groupingCode) => + this.histogramSplitOptions(block).find((option) => option.code === groupingCode)?.label ?? groupingCode, + }; + }, + pendingPreprocessing: () => this.buildPreprocessingConfig(this.pendingPreprocessingRules, this.currentPreprocessingCodeSet()), + rawPreviewFilter: () => this.rawPreviewFilter, + groupLabel: (groupingVar, groupingEnum) => this.groupingEnumLabel(groupingVar, groupingEnum), + changed: () => this.cdr.markForCheck(), + }; effect(() => { const layout = getAnalysisGuideLayout(this.guideState.activeStepId()); if (!layout) return; @@ -1160,8 +942,8 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const filters = this.expStudioService.selectedFilters(); const filterLogic = this.expStudioService.filterLogic(); const appliedPreprocessing = this.expStudioService.appliedPreprocessingConfig(); - const cohortKey = this.buildSelectionKey(variables, filters, filterLogic); - const nextSelectionKey = this.buildSelectionKey(variables, filters, filterLogic, appliedPreprocessing); + const cohortKey = selectionKey(variables, filters, filterLogic); + const nextSelectionKey = selectionKey(variables, filters, filterLogic, appliedPreprocessing); const echoed = this.appliedConfigWrite; if ( echoed @@ -1185,12 +967,9 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.sourceDescribeSeq++; this.clearProcessedPreviewSubscription(); this.clearSourcePreviewSubscription(); - this.clearHistogramSubscriptions(this.rawSummary); - this.clearHistogramSubscriptions(this.sourceSummary); - this.clearHistogramSubscriptions(this.processedSummary); - this.rawSummary = this.createEmptySummary(false); - this.processedSummary = this.createEmptySummary(false); - this.sourceSummary = this.createEmptySummary(false); + this.rawSummary = emptySummaryView(false); + this.processedSummary = emptySummaryView(false); + this.sourceSummary = emptySummaryView(false); this.rawSummaryKey = ''; this.sourceSummaryKey = ''; this.processedSummaryKey = ''; @@ -1212,19 +991,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { // not become a dependency of this effect or every section switch refetches raw. if (untracked(this.sectionOpen).source) this.ensureSourceSummary(); }); - - effect(() => { - const request = this.scrollSectionRequest(); - if (!request) return; - - setTimeout(() => { - if (this.scrollSectionRequest()?.requestId !== request.requestId) return; - this.scrollSectionIntoView(request.section); - if (this.scrollSectionRequest()?.requestId === request.requestId) { - this.scrollSectionRequest.set(null); - } - }); - }); } ngOnDestroy(): void { @@ -1232,9 +998,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.destroyed.complete(); this.clearProcessedPreviewSubscription(); this.clearSourcePreviewSubscription(); - this.clearHistogramSubscriptions(this.rawSummary); - this.clearHistogramSubscriptions(this.sourceSummary); - this.clearHistogramSubscriptions(this.processedSummary); } get pendingChangeCount(): number { @@ -1245,49 +1008,24 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { countFilterRules(this.expStudioService.filterLogic()) ); - /** Drafts that already form a valid column config (code + at least one filtered rule). */ - private transformationConfigs(): Record[] { - return this.transformationDrafts - .map((draft) => this.buildTransformationConfigForDraft(draft)) - .filter((config): config is Record => !!config); - } - - /** A draft counts as touched once the user typed any column name, default, or rule. */ - private draftHasWork(draft: TransformationColumnDraft): boolean { - return ( - draft.code.trim().length > 0 || - draft.defaultEnumeration.trim().length > 0 || - draft.rules.some((rule) => rule.value.trim().length > 0 || !!rule.filter) - ); - } - /** Two columns cannot share a name: exaflow errors when `code` already exists. */ get transformationHasDuplicateCodes(): boolean { - const codes = this.transformationDrafts.map((draft) => draft.code.trim()).filter((code) => code.length > 0); - return new Set(codes).size !== codes.length; + return this.transformationDraftState.hasDuplicateCodes; } get transformationHasPendingChange(): boolean { - if (this.transformationHasDuplicateCodes) return true; - return this.transformationDrafts.some( - (draft) => this.draftHasWork(draft) && (this.hasUnfinishedCategory(draft) || !this.buildTransformationConfigForDraft(draft)) - ); - } - - /** A category without a name or a rule is left out of the saved column. */ - private hasUnfinishedCategory(draft: TransformationColumnDraft): boolean { - return draft.rules.some((rule) => !rule.value.trim() || !rule.filter); + return this.transformationDraftState.hasPendingChange; } get transformationStatusLabel(): string { - const validCount = this.transformationConfigs().length; + const validCount = this.transformationDraftState.configs().length; const hasAppliedColumn = validCount > 0 || this.cards().some((step) => step.name !== 'categorical_column_creator' && stepOutputNamesOf(step).length > 0); if (hasAppliedColumn && !this.transformationHasPendingChange) return 'Applied'; if ( hasAppliedColumn || this.transformationHasPendingChange - || this.transformationDrafts.some((draft) => this.draftHasWork(draft)) + || this.transformationDraftState.hasAnyWork || this.cards().length > 0 ) { return 'Not applied'; @@ -1305,49 +1043,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } - /** Status chip for the K-means cluster column card on the Transformation step. */ - get kmeansClusterStatusLabel(): string { - return this.cards().some( - (step) => step.name === 'kmeans_cluster_creator' && stepOutputNamesOf(step).length > 0, - ) - ? 'Applied' - : 'Not set'; - } - - /** True when a named category on any card is still missing an applied filter. */ - get transformationRulesNeedFilters(): boolean { - return this.transformationDrafts.some((draft) => { - if (!draft.code.trim()) return false; - const named = draft.rules.filter((rule) => rule.value.trim().length > 0); - return named.length > 0 && named.some((rule) => !rule.filter); - }); - } - get pendingMissingChangeCount(): number { return this.hasPendingMissingChanges ? 1 : 0; } get hasPendingMissingChanges(): boolean { - const currentCodes = this.currentPreprocessingCodeSet(); - const codes = new Set( - [ - ...Array.from(currentCodes), - ...Object.keys(this.pendingPreprocessingRules), - ...Object.keys(this.appliedPreprocessingRules), - ].filter((code) => currentCodes.has(code)) - ); - for (const code of codes) { - const appliedRule = this.appliedPreprocessingRules[code]; - const pendingRule = this.pendingPreprocessingRules[code] - ?? (appliedRule ? undefined : this.defaultRule(code)); - // Newly added variables sit on the implicit NA-removal default; that - // doesn't count as a pending change because no user action is required. - if (this.isDefaultMissingRule(pendingRule) && !this.hasAppliedPreprocessing(code)) continue; - if (this.serializeRule(pendingRule) !== this.serializeRule(appliedRule)) { - return true; - } - } - return false; + return this.preprocessingVariables.some((variable) => this.preprocessingState.missingHasPendingChange(variable.code)); } get pendingOutlierChangeCount(): number { @@ -1355,22 +1056,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } get hasPendingOutlierChanges(): boolean { - const currentCodes = this.currentOutlierPreprocessingCodeSet(); - const codes = new Set( - [ - ...Array.from(currentCodes), - ...Object.keys(this.pendingOutlierRules), - ...Object.keys(this.appliedOutlierRules), - ].filter((code) => currentCodes.has(code)) - ); - for (const code of codes) { - const pendingRule = this.pendingOutlierRules[code] - ?? (this.appliedOutlierRules[code] ? undefined : this.defaultOutlierRule(code)); - if (serializeOutlierRule(pendingRule) !== serializeOutlierRule(this.appliedOutlierRules[code])) { - return true; - } - } - return false; + return this.outlierPreprocessingVariables.some((variable) => this.preprocessingState.outlierHasPendingChange(variable.code)); } get pendingLongitudinalChangeCount(): number { @@ -1400,55 +1086,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } prepGroups(kind: PrepKind): PreprocessingGroup[] { - return this.buildPreprocessingGroups( - this.filteredPrepVariables(kind), - (variable) => this.prepVariableHasPendingChange(kind, variable), - (variable) => this.prepVariableIsApplied(kind, variable), - kind === 'missing' ? (variable) => this.isPreprocessingVariableUsingDefault(variable) : () => false, - ); - } - - private buildPreprocessingGroups( - variables: VariableRow[], - isPending: (variable: VariableRow) => boolean, - isApplied: (variable: VariableRow) => boolean, - isDefault: (variable: VariableRow) => boolean = () => false - ): PreprocessingGroup[] { - const pending: VariableRow[] = []; - const applied: VariableRow[] = []; - const defaults: VariableRow[] = []; - const notApplied: VariableRow[] = []; - - variables.forEach((variable) => { - if (isPending(variable)) pending.push(variable); - else if (isApplied(variable)) applied.push(variable); - else if (isDefault(variable)) defaults.push(variable); - else notApplied.push(variable); + return groupPreprocessingVariables(this.filteredPrepVariables(kind), (variable) => { + if (this.prepVariableHasPendingChange(kind, variable)) return 'pending'; + if (this.prepVariableIsApplied(kind, variable)) return 'applied'; + if (kind === 'missing' && this.isPreprocessingVariableUsingDefault(variable)) return 'default'; + return 'not-applied'; }); - - const groups: PreprocessingGroup[] = [ - { - key: 'pending', - title: 'Pending', - variables: pending, - }, - { - key: 'applied', - title: 'Applied preprocessing', - variables: applied, - }, - { - key: 'default', - title: 'Default (NA removal)', - variables: defaults, - }, - { - key: 'not-applied', - title: 'Not applied', - variables: notApplied, - }, - ]; - return groups.filter((group) => group.variables.length > 0); } selectedPrepVariable(kind: PrepKind): VariableRow | null { @@ -1473,23 +1116,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return 'Not applied'; } - prepVariableHasPendingChange(kind: PrepKind, variable: VariableRow): boolean { + prepVariableHasPendingChange(kind: PrepKind, variable: VariableRow): boolean { switch (kind) { - case 'missing': { - const appliedRule = this.appliedPreprocessingRules[variable.code]; - const pendingRule = this.pendingPreprocessingRules[variable.code] - ?? (appliedRule ? undefined : this.defaultRule(variable.code)); - if (this.isDefaultMissingRule(pendingRule) && !this.hasAppliedPreprocessing(variable.code)) { - return false; - } - return this.serializeRule(pendingRule) !== this.serializeRule(appliedRule); - } - case 'outlier': { - const appliedRule = this.appliedOutlierRules[variable.code]; - const pendingRule = this.pendingOutlierRules[variable.code] - ?? (appliedRule ? undefined : this.defaultOutlierRule(variable.code)); - return serializeOutlierRule(pendingRule) !== serializeOutlierRule(appliedRule); - } + case 'missing': + return this.preprocessingState.missingHasPendingChange(variable.code); + case 'outlier': + return this.preprocessingState.outlierHasPendingChange(variable.code); case 'longitudinal': { if (!this.isLongitudinalModel) return false; const pendingStrategy = this.pendingLongitudinalStrategies[variable.code] ?? this.defaultLongitudinalStrategy(variable); @@ -1519,17 +1151,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } isPreprocessingVariableUsingDefault(variable: VariableRow): boolean { - if (this.hasAppliedPreprocessing(variable.code)) return false; - const pendingRule = this.pendingPreprocessingRules[variable.code] - ?? this.defaultRule(variable.code); - return this.isDefaultMissingRule(pendingRule); - } - - private isDefaultMissingRule(rule: PreprocessingRule | undefined): boolean { - if (!rule) return false; - return rule.enabled === true - && rule.action === 'drop' - && (rule.value === undefined || rule.value === ''); + return this.preprocessingState.isUsingDefaultMissing(variable.code); } get longitudinalVisitPairHasPendingChange(): boolean { @@ -1594,7 +1216,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.summaryExportMenuKind.set(null); if (tab === 'Histogram') { const block = this.selectedStatisticBlock(kind); - if (block) this.ensureHistogramForBlock(kind, block); + if (block) this.summaryHistogram.ensure(kind, block); } this.cdr.markForCheck(); } @@ -1620,9 +1242,9 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { selectStatisticBlock(kind: SummaryKind, block: PivotBlock): void { const summary = this.getSummary(kind); - summary.selectedStatisticKey = this.statisticBlockKey(block); + summary.selectedStatisticKey = statisticBlockKey(block); if (summary.activeTab === 'Histogram') { - this.ensureHistogramForBlock(kind, block); + this.summaryHistogram.ensure(kind, block); } } @@ -1630,12 +1252,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const filtered = this.filteredStatisticBlocks(kind); if (!filtered.length) return null; const selectedKey = this.getSummary(kind).selectedStatisticKey; - return filtered.find((block) => this.statisticBlockKey(block) === selectedKey) ?? filtered[0]; + return filtered.find((block) => statisticBlockKey(block) === selectedKey) ?? filtered[0]; } isStatisticBlockSelected(kind: SummaryKind, block: PivotBlock): boolean { const selected = this.selectedStatisticBlock(kind); - return !!selected && this.statisticBlockKey(selected) === this.statisticBlockKey(block); + return !!selected && statisticBlockKey(selected) === statisticBlockKey(block); } statisticBlockVariable(block: PivotBlock): VariableRow | null { @@ -1650,10 +1272,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return this.statisticBlockType(kind, block); } - selectedSummaryChartTypeLabel(kind: SummaryKind, block: PivotBlock): string { - return this.selectedSummaryChartType(kind, block) === 'numeric' ? 'Numerical' : 'Categorical'; - } - selectedSummaryChartOptions(kind: SummaryKind, block: PivotBlock): EChartsOption[] { const variable = this.statisticBlockVariable(block); if (!variable) return []; @@ -1726,7 +1344,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { ...summary.histogramSplitByVariable, [code]: split, }; - this.ensureHistogramForBlock(kind, block); + this.summaryHistogram.ensure(kind, block); this.cdr.markForCheck(); } @@ -1746,31 +1364,11 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } goToSection(section: SectionKey): void { - const filtersStation = section === 'filters' || section === 'raw'; - const setupStation = section === 'setup' || section === 'processed'; - const previewFilters = section === 'raw'; - const previewSetup = section === 'processed'; - if (filtersStation) { - this.addedSteps.update((s) => ({ ...s, filters: true })); - } - if (setupStation) { - this.addedSteps.update((s) => ({ ...s, setup: true })); - } - if (section === 'transformation') { - this.addedSteps.update((s) => ({ ...s, transformation: true })); - } - this.summaryExpanded.set({ - filters: previewFilters, - setup: previewSetup, - }); - this.sectionOpen.set({ - source: section === 'source', - filters: filtersStation && !previewFilters, - raw: previewFilters, - setup: setupStation && !previewSetup, - processed: previewSetup, - transformation: section === 'transformation', - }); + const layout = sectionLayout(section); + const addsStep = layout.addsStep; + if (addsStep) this.addedSteps.update((s) => ({ ...s, [addsStep]: true })); + this.summaryExpanded.set(layout.summaryExpanded); + this.sectionOpen.set(layout.sectionOpen); if (section === 'raw') { // The Raw summary must show the cohort it claims to: the stored rules, or the // unsaved ones a Cohort Filtering preview pinned to it. @@ -1783,16 +1381,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.fetchProcessedPreview(); } this.cdr.markForCheck(); - this.requestSectionScroll(section); - } - - /** - * Scrolling is a request, not a call: the section is measured only after the template - * has reacted to whatever just opened or closed. Shared by goToSection and the stations' - * Close action, which folds an editor away and must not leave the viewport stranded. - */ - private requestSectionScroll(section: SectionKey): void { - this.scrollSectionRequest.set({ section, requestId: ++this.scrollRequestId }); + this.sectionAnchors.requestScroll(section); } /** @@ -1838,7 +1427,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const variableCodes = this.preprocessingVariables.map((item) => item.code); const hasScope = !!this.expStudioService.selectedDataModel() && this.expStudioService.selectedDatasets().length > 0; if (!hasScope || !variableCodes.length) { - this.sourceSummary = this.createEmptySummary(false); + this.sourceSummary = emptySummaryView(false); this.sourceSummaryKey = ''; this.cdr.markForCheck(); return; @@ -1846,8 +1435,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const requestedKey = this.sourceScopeKey; const requestSeq = ++this.sourceDescribeSeq; - this.clearHistogramSubscriptions(this.sourceSummary); - this.sourceSummary = { ...this.sourceSummary, isLoading: true }; + this.sourceSummary = { ...this.sourceSummary, isLoading: true, histogramSettledByVariable: {} }; this.cdr.markForCheck(); // No filter payload, no preprocessing steps: the data exactly as selected. @@ -1861,7 +1449,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.clearSourcePreviewSubscription(); this.sourceSummary = this.buildSummaryFromResponse(response, 'source'); this.sourceSummaryKey = requestedKey; - this.refreshActiveHistogramPreview('source'); + this.summaryHistogram.refreshActive('source'); this.cdr.markForCheck(); }, error: (err) => { @@ -1874,9 +1462,8 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { }); } - /** Heading and aria text, so one summary template reads correctly for all three kinds. */ summaryKindLabel(kind: SummaryKind): string { - return kind === 'source' ? 'Source data' : kind === 'raw' ? 'Raw data' : 'Processed data'; + return summaryKindLabel(kind); } /** Anchor name for one slot of the shared summary template; see `SUMMARY_GUIDE_ANCHORS`. */ @@ -1895,14 +1482,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } collapseAllWorkflowSections(): void { - this.sectionOpen.set({ - source: false, - raw: false, - setup: false, - filters: false, - processed: false, - transformation: false, - }); + this.sectionOpen.set({ ...CLOSED_SECTIONS }); this.cdr.markForCheck(); } @@ -1929,28 +1509,22 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } /** - * Persist live category-filter builders onto every rule (across all cards) without - * touching cohort filters. `#ruleModal` renders one modal per rule in DOM order, so a - * single running index walks drafts and their rules in the same order the template emits. - * A rule is rewritten only when its builder can speak for it: a builder that rejects its - * own half-typed condition, or that could not read what the store holds, leaves that rule - * as it is rather than exporting an empty filter and erasing a configured category. + * Persist live category-filter builders onto every rule (across all cards, or only + * `only`'s) without touching cohort filters. A rule is rewritten only when its builder + * can speak for it: a builder that rejects its own half-typed condition, or that could + * not read what the store holds, leaves that rule as it is rather than exporting an + * empty filter and erasing a configured category. */ commitTransformationRuleFilters(only?: TransformationColumnDraft): void { - const modals = this.transformationRuleModals?.toArray() ?? []; - let modalIndex = 0; for (const draft of this.transformationDrafts) { + if (only && draft !== only) continue; for (const rule of draft.rules) { - const modal = modals[modalIndex++]; - if (!modal || (only && draft !== only)) { - continue; - } - const logic = modal.exportFilterLogic(); - if (modal.filterError()) { - // Keep walking the remaining cards: one broken builder must not stop the - // others from committing what they hold. The builder renders its own error. - continue; - } + const builder = this.ruleBuilders.get(rule); + if (!builder) continue; + const logic = builder.exportFilterLogic(); + // Keep walking the remaining cards: one broken builder must not stop the + // others from committing what they hold. The builder renders its own error. + if (builder.filterError()) continue; rule.filter = logic; } } @@ -1962,7 +1536,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.previewDraftId = stepId ?? this.cards().find((step) => step.name === 'kmeans_cluster_creator')?.id ?? null; - this.refreshTransformationStatistics(); + this.transformationStats.refresh(); this.cdr.markForCheck(); } @@ -1976,17 +1550,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { else this.previewTransformationCard(); } - transformationStatsFor(code: string): Array<{ code: string; rows: Array<{ value: string; count: number | null }> }> { - const target = code.trim(); - return this.transformationStatistics.filter((block) => block.code === target); - } - - /** Card is incomplete or clashes: its primary slot says Apply, otherwise Close. */ - draftHasNewWork(draft: TransformationColumnDraft): boolean { - const { label } = this.transformationDraftStatus(draft); - return label === 'Not applied' || label === 'Duplicate name'; - } - /** A card's own Apply/Close: commit its rule builders, then fold it once it is settled. * Valid cards are persisted as they are typed, so this never writes other cards. */ commitTransformationDraft(draft: TransformationColumnDraft): void { @@ -1996,39 +1559,31 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const rule = draft.rules[index]; if (!rule.value.trim() && !rule.filter) this.removeTransformationRule(draft, index); } - if (!this.draftHasNewWork(draft)) draft.open = false; + if (!this.transformationDraftState.hasNewWork(draft)) draft.open = false; this.cdr.markForCheck(); } /** Clear the transformation station: pending edits and the committed config alike. */ resetTransformation(): void { - this.expStudioService.clearTransformations(); - this.transformationDrafts = []; - this.cardOpenState.set({}); + this.transformationDraftState.reset(); this.previewDraftId = null; - this.transformationStatistics = []; - this.transformationStatisticsError = ''; - this.invalidateCategoryCounts(); + this.transformationStats.clear(); + this.transformationStats.invalidate(); this.emitProgressState(); this.cdr.markForCheck(); } /** A new category opens straight onto its filter, with the first condition ready. */ addTransformationRule(draft: TransformationColumnDraft): void { - draft.rules.push({ value: '', filter: null }); - const index = draft.rules.length - 1; - draft.openRuleIndex = index; + const index = this.transformationDraftState.addRule(draft); this.onTransformationChange(); this.cdr.detectChanges(); - this.ruleModalAt(draft, index)?.ensureFirstCondition(); + this.ruleBuilderAt(draft, index)?.ensureFirstCondition(); } - /** `#ruleModal` renders one builder per rule in card order, so earlier cards' rules come first. */ - private ruleModalAt(draft: TransformationColumnDraft, index: number) { - const offset = this.transformationDrafts - .slice(0, this.transformationDrafts.indexOf(draft)) - .reduce((count, earlier) => count + earlier.rules.length, 0); - return this.transformationRuleModals?.toArray()[offset + index]; + private ruleBuilderAt(draft: TransformationColumnDraft, index: number): RuleBuilder | undefined { + const rule = draft.rules[index]; + return rule ? this.ruleBuilders.get(rule) : undefined; } /** @@ -2036,7 +1591,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { * category list. A builder with an error stays open and shows its own message. */ finishTransformationRule(draft: TransformationColumnDraft, index: number): void { - const modal = this.ruleModalAt(draft, index); + const modal = this.ruleBuilderAt(draft, index); if (modal) { const logic = modal.exportFilterLogic(); if (modal.filterError()) { @@ -2047,14 +1602,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } draft.openRuleIndex = null; this.onTransformationChange(); - this.countsDraftId = draft.id; - this.refreshTransformationStatistics(); + this.transformationStats.countsDraftId.set(draft.id); + this.transformationStats.refresh(); } removeTransformationRule(draft: TransformationColumnDraft, index: number): void { - draft.rules.splice(index, 1); - if (draft.openRuleIndex === index) draft.openRuleIndex = null; - else if (draft.openRuleIndex != null && draft.openRuleIndex > index) draft.openRuleIndex -= 1; + this.transformationDraftState.removeRule(draft, index); this.onTransformationChange(); } @@ -2063,22 +1616,9 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.cdr.markForCheck(); } - private firstFilterCondition(filter: unknown): { field: string; operator: string; value: unknown } | null { - if (!filter || typeof filter !== 'object') return null; - const node = filter as { field?: unknown; operator?: unknown; value?: unknown; rules?: unknown[] }; - if (node.field) { - return { field: String(node.field), operator: String(node.operator ?? ''), value: node.value }; - } - for (const rule of Array.isArray(node.rules) ? node.rules : []) { - const found = this.firstFilterCondition(rule); - if (found) return found; - } - return null; - } - /** A one-line rule summary for the collapsed row. */ transformationRuleSummary(rule: TransformationRule): string { - const condition = this.firstFilterCondition(rule.filter); + const condition = firstFilterCondition(rule.filter); return condition ? this.filterConditionSentence(condition) : 'No rule yet'; } @@ -2096,62 +1636,18 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return values ? `${label} ${phrase} ${values}` : `${label} ${phrase}`; } - /** Loaded record count for one enumeration of a card's column. */ - transformationCount(draft: TransformationColumnDraft, value: string): string { - const block = this.transformationStatistics.find((item) => item.code === draft.code.trim()); - const row = block?.rows.find((item) => item.value === value); - return row?.count != null ? row.count.toLocaleString() : '—'; - } - - /** Per-card chip: Applied only when every category is complete and the name is unique. - * An unfinished row is omitted from the saved column, so it must not leave the chip on Applied. */ - transformationDraftStatus(draft: TransformationColumnDraft): { label: string; tone: 'applied' | 'pending' | 'default' } { - const problem = this.draftWriteProblem(draft); - if (problem) return { label: problem === 'Name already used' ? 'Duplicate name' : 'Column in use', tone: 'pending' }; - if (!this.hasUnfinishedCategory(draft) && this.buildTransformationConfigForDraft(draft)) { - return { label: 'Applied', tone: 'applied' }; - } - if (this.draftHasWork(draft)) return { label: 'Not applied', tone: 'pending' }; - return { label: 'Not set', tone: 'default' }; - } - - /** - * Why a categorical card cannot be written to the store: a name collision, or a later step - * still reads the column the edit would drop (renamed, or left incomplete). Mirrors the - * refusal in `upsertTransformation`, so a refused edit is shown on the card instead of lost. - */ - private draftWriteProblem(draft: TransformationColumnDraft): string | null { - const code = draft.code.trim(); - const nameError = code ? this.columnNameError(draft.id, code) : null; - if (nameError || this.buildTransformationConfigForDraft(draft)) return nameError; - const inUse = this.expStudioService.outputsInUse(draft.id, []); - return inUse.length ? `Used by a later step: ${inUse.join(', ')}` : null; - } - toggleTransformationDraft(draft: TransformationColumnDraft): void { draft.open = !draft.open; this.cdr.markForCheck(); } - /** Append another empty categorical card through the store-backed chooser path. */ - addTransformationDraft(): void { - this.chooseTransformation('categorical_column_creator'); - } - /** * Remove a store step (and its categorical draft). Disabled while dependents remain — * callers should check `canRemoveTransformation` first. */ removeTransformationDraft(id: string): void { - if (!this.canRemoveTransformation(id)) return; - this.expStudioService.removeTransformation(id); - this.transformationDrafts = this.transformationDrafts.filter((draft) => draft.id !== id); - this.cardOpenState.update((state) => { - const next = { ...state }; - delete next[id]; - return next; - }); - this.invalidateCategoryCounts(); + if (!this.transformationDraftState.remove(id)) return; + this.transformationStats.invalidate(); this.emitProgressState(); this.cdr.markForCheck(); } @@ -2171,82 +1667,9 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } formatPreprocessingDocumentationHtml(stepName: string): string { - return this.buildPreprocessingDocumentationHtml(this.preprocessingStepDocumentation(stepName)); + return buildPreprocessingDocumentationHtml(this.preprocessingStepDocumentation(stepName)); } - private buildPreprocessingDocumentationHtml(text: string): string { - const trimmed = text.trim(); - if (!trimmed) { - return ''; - } - - const lines = trimmed.includes('\n') - ? trimmed.split(/\r?\n/) - : trimmed.split(/\s+(?:[-*\u2022\u00b7])\s+(?=[A-Za-z])/); - - const parts: string[] = []; - let listItems: string[] = []; - let isIntroParagraph = true; - - const flushList = (): void => { - if (!listItems.length) { - return; - } - parts.push('
    ' + listItems.join('') + '
'); - listItems = []; - }; - - const pushParagraph = (line: string): void => { - const escaped = escapeHtml(line); - if (line.endsWith(':') && line.length <= 120) { - parts.push('

' + escaped + '

'); - isIntroParagraph = false; - return; - } - - const classes = isIntroParagraph - ? 'preprocessing-doc-paragraph preprocessing-doc-intro' - : 'preprocessing-doc-paragraph'; - parts.push('

' + escaped + '

'); - isIntroParagraph = false; - }; - - for (const rawLine of lines) { - const line = rawLine.trim(); - if (!line) { - continue; - } - - const isBullet = DOCUMENTATION_BULLET.test(line); - let itemText = isBullet ? line.replace(DOCUMENTATION_BULLET, '').trim() : line; - const colonIndex = itemText.indexOf(':'); - const looksLikeListItem = - isBullet || - (colonIndex > 0 && colonIndex < 40 && itemText.slice(colonIndex + 1).trim().length > 0); - - if (looksLikeListItem && colonIndex > 0) { - const term = escapeHtml(itemText.slice(0, colonIndex).trim()); - const description = escapeHtml(itemText.slice(colonIndex + 1).trim()); - // The term keeps its colon: the item is read as one line of prose, not a table. - listItems.push( - '
  • ' + term + ': ' + - '' + description + '
    • ' - ); - continue; - } - - if (isBullet) { - listItems.push('
  • ' + escapeHtml(itemText) + '
    • '); - continue; - } - - flushList(); - pushParagraph(line); - } - - flushList(); - return parts.join(''); - } // ── Preprocessing evidence ──────────────────────────────────────────────── // A preprocessing form states a decision but never priced it. These read the @@ -2258,57 +1681,26 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return this.rawSummary.data.find((block) => this.variableCodeForBlock(block) === variable.code) ?? null; } - coverageFor(variable: VariableRow): { total: number; missing: number; share: number } | null { - return this.blockCoverage(this.summaryBlockFor(variable)); + coverageFor(variable: VariableRow): Coverage | null { + return blockCoverage(this.summaryBlockFor(variable)); } - /** Block-based coverage. Both rails price a variable the same way, so the - * lookup lives here and coverageFor() stays the variable-based entry. */ - blockCoverage(block: PivotBlock | null | undefined): { total: number; missing: number; share: number } | null { - if (!block) return null; - const total = this.statisticMetricNumber(block, 'Total'); - if (total <= 0) return null; - const missing = Math.min(this.statisticMissingValue(block), total); - return { total, missing, share: missing / total }; + /** Both rails price a variable the same way; coverageFor() is the variable-based entry. */ + blockCoverage(block: PivotBlock | null | undefined): Coverage | null { + return blockCoverage(block); } - /** Display form of a canonical pivot value: parse the stored string, then - * locale-group it. Storage and CSV export use `formatNumber`, never this. */ displayNumber(value: string | undefined | null): string { - if (value === null || value === undefined || value === '') return value ?? ''; - const n = Number(String(value).replace(/,/g, '')); - if (!Number.isFinite(n)) return value; - return n.toLocaleString(undefined, { maximumFractionDigits: 2 }); + return displayNumber(value); } missingShareNote(block: PivotBlock, row: PivotBlock['rows'][number], dataset: string): string | null { - if (row.metric !== 'Missing') return null; - const parse = (metric: string) => - Number((block.rows.find((item) => item.metric === metric)?.values[dataset] ?? '').replace(/,/g, '')); - const missing = parse('Missing'); - const total = parse('Total'); - if (!Number.isFinite(missing) || !Number.isFinite(total) || total <= 0) return null; - return this.coveragePercent({ share: Math.min(missing / total, 1) }); - } - - /** The rail's foot: coverage re-pooled over every variable currently shown. - * Per-variable shares are not additive, so this sums the counts and divides - * once — averaging 0.7% and 100% would state a number that describes no - * dataset. Null when no shown block has statistics, so the foot disappears - * rather than reporting 0%. */ - selectionCoverage(kind: SummaryKind): { total: number; missing: number; share: number; variables: number } | null { - let total = 0; - let missing = 0; - let variables = 0; - for (const block of this.filteredStatisticBlocks(kind)) { - const coverage = this.blockCoverage(block); - if (!coverage) continue; - total += coverage.total; - missing += coverage.missing; - variables += 1; - } - if (variables === 0 || total <= 0) return null; - return { total, missing, share: Math.min(missing / total, 1), variables }; + return missingShareNote(block, row, dataset); + } + + /** The rail's foot: coverage re-pooled over every variable currently shown. */ + selectionCoverage(kind: SummaryKind): PooledCoverage | null { + return pooledCoverage(this.filteredStatisticBlocks(kind)); } /** The pooled column is a rollup, not another dataset. Normalised through the @@ -2318,33 +1710,20 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return this.datasetLabel(column) === this.datasetLabel('all datasets'); } - private isCountMetric(metric: string | undefined): boolean { - return metric === 'Datapoints' || metric === 'Missing' || metric === 'Total'; - } - - /** Group heading rendered above rows[index], or null when the row continues - * the group above it. A group starts where count-ness changes. */ metricGroupLabel(rows: PivotBlock['rows'], index: number): string | null { - const here = this.isCountMetric(rows[index]?.metric); - if (index === 0 || this.isCountMetric(rows[index - 1]?.metric) !== here) { - return here ? 'Coverage' : 'Distribution'; - } - return null; + return metricGroupLabel(rows, index); } coveragePercent(coverage: { share: number }): string { - return this.formatPercent(coverage.share * 100); + return coveragePercent(coverage); } - /** Bar length for a share scaled up so small shares stay visible (×1000: 5% is a half - * bar), floored at 2% so a clean variable still shows the track. */ barWidth(share: unknown, scale: number): number { - return Math.max(2, Math.min(100, (parseNumber(share) ?? 0) * scale)); + return barWidth(share, scale); } - /** The full count belongs in its tooltip; the row only carries the share. */ coverageTitle(coverage: { total: number; missing: number }): string { - return `${coverage.missing.toLocaleString()} of ${coverage.total.toLocaleString()} values missing`; + return coverageTitle(coverage); } /** An unpriced decision must say why it is unpriced, not render blank. @@ -2355,52 +1734,16 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { : 'No dataset statistics are available for this variable, so neither the distribution nor the row cost can be shown.'; } - /** Three bands. At the old single 10% floor 30% and 100% rendered in the same - * red, so across a rail of variables the colour ranked nothing. 10% keeps the - * studio's existing "worth noticing" onset; 50% is where a variable stops - * being usable. The exact figure always sits beside the colour. */ coverageSeverity(coverage: { share: number }): 'none' | 'warning' | 'high' { - if (coverage.share >= 0.5) return 'high'; - if (coverage.share >= 0.1) return 'warning'; - return 'none'; - } - - distributionFor(variable: VariableRow): Array<{ label: string; value: string }> { - const block = this.summaryBlockFor(variable); - if (!block) return []; - // A nominal block carries no quantile rows at all, so the lookups come back - // 'N/A'; those are absent data, not values, and must not render as a row of N/A. - return (['mean', 'std', 'min', 'q1', 'q2', 'q3', 'max'] as const) - .map((key) => ({ - label: this.metricLabel[key], - value: this.displayNumber(this.statisticMetricText(block, this.metricLabel[key])), - })) - .filter((point) => point.value !== 'N/A'); - } - - /** Category frequencies. In the pivot these are the rows after the three - * count metrics, so anything that is not a count metric is a category — but - * only for a nominal block. A numeric block's remaining rows are Mean/Std/ - * Min/Q1/Q2/Q3/Max, which are not categories and were being relabelled. */ - categoryCountsFor(variable: VariableRow, limit = 6): Array<{ label: string; value: string; share: number }> { - if (!this.isCategoricalVariable(variable)) return []; - const block = this.summaryBlockFor(variable); - if (!block) return []; - const base = new Set(['Datapoints', 'Missing', 'Total']); - const dataset = block.columns.includes('all datasets') ? 'all datasets' : block.columns[0]; - const datapoints = Number((block.rows.find((row) => row.metric === 'Datapoints')?.values[dataset] ?? '').replace(/,/g, '')); - return block.rows - .filter((row) => !base.has(row.metric)) - .map((row) => { - const value = row.values[dataset] ?? ''; - const count = Number(value.replace(/,/g, '')); - return { - label: row.metric, - value: this.displayNumber(value || '0'), - share: Number.isFinite(count) && datapoints > 0 ? Math.min(count / datapoints, 1) : 0, - }; - }) - .slice(0, limit); + return coverageSeverity(coverage); + } + + distributionFor(variable: VariableRow): DistributionPoint[] { + return blockDistribution(this.summaryBlockFor(variable)); + } + + categoryCountsFor(variable: VariableRow, limit = 6): CategoryShare[] { + return this.isCategoricalVariable(variable) ? categoryShares(this.summaryBlockFor(variable), limit) : []; } /** The selected missing-value action restated in rows, so it can be judged. */ @@ -2531,35 +1874,13 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { .find((point) => point.label.toLowerCase() === label.toLowerCase())?.value ?? ''; } - /** Per-dataset coverage for the missing-values evidence tiles. Derives the same - * Missing/Total rows the table uses, split by dataset, minus the rollup column. */ - coverageByDataset(variable: VariableRow): Array<{ label: string; missing: number; total: number; share: number }> { - const block = this.summaryBlockFor(variable); - if (!block) return []; - const totalRow = block.rows.find((row) => row.metric === 'Total'); - const missingRow = block.rows.find((row) => row.metric === 'Missing'); - if (!totalRow || !missingRow) return []; - return block.columns - .filter((dataset) => !this.isRollupColumn(dataset)) - .map((dataset) => { - const total = parseNumber(totalRow.values[dataset]) ?? 0; - if (total <= 0) return null; - const missing = Math.min(parseNumber(missingRow.values[dataset]) ?? 0, total); - return { - label: this.datasetLabel(dataset), - missing, - total, - share: missing / total, - }; - }) - .filter((tile): tile is { label: string; missing: number; total: number; share: number } => !!tile); + /** Per-dataset coverage for the missing-values evidence tiles, minus the rollup column. */ + coverageByDataset(variable: VariableRow): DatasetCoverage[] { + return coverageByDataset(this.summaryBlockFor(variable), (code) => this.datasetLabel(code)); } ruleFor(variable: VariableRow): PreprocessingRule { - if (!this.pendingPreprocessingRules[variable.code]) { - this.pendingPreprocessingRules[variable.code] = this.defaultRule(variable.code); - } - return this.pendingPreprocessingRules[variable.code]; + return this.preprocessingState.missingRule(variable.code); } onMissingActionChange(variable: VariableRow, action: MissingAction): void { @@ -2637,108 +1958,16 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return this.outlierPreviewRows.filter((row) => row.variable === label || row.variable === variable.code); } - private outlierDistributionNumbers(variable: VariableRow): { - min: number; q1: number; median: number; q3: number; max: number; mean: number | null; std: number | null; - } | null { - const points = this.distributionFor(variable); - if (!points.length) return null; - const byLabel = new Map(points.map((point) => [point.label.toLowerCase(), point.value])); - const number = (label: string) => parseNumber(byLabel.get(label)); - const min = number('min'); - const q1 = number('q1'); - const median = number('median'); - const q3 = number('q3'); - const max = number('max'); - if (min === null || q1 === null || median === null || q3 === null || max === null) return null; - return { min, q1, median, q3, max, mean: number('mean'), std: number('standard deviation') }; - } - - /** Client-side bounds for the strategies the summary can price. MAD/quantile and - * Gaussian without mean/std return null; the strip then says so. */ - private outlierBounds(variable: VariableRow): { lo: number | null; hi: number | null } | null { - const numbers = this.outlierDistributionNumbers(variable); - if (!numbers) return null; - const rule = this.outlierRuleFor(variable); - const k = Number(rule.fold); - if (!Number.isFinite(k)) return null; - let lo: number; - let hi: number; - switch (rule.strategy) { - case 'iqr': { - const iqr = numbers.q3 - numbers.q1; - lo = numbers.q1 - k * iqr; - hi = numbers.q3 + k * iqr; - break; - } - case 'gaussian': { - if (numbers.mean === null || numbers.std === null) return this.reportBounds(variable); - lo = numbers.mean - k * numbers.std; - hi = numbers.mean + k * numbers.std; - break; - } - default: - // MAD and arbitrary quantiles are not in the summary; the report's own - // bounds are the fallback once a report has run. - return this.reportBounds(variable); - } - return { - lo: rule.tail === 'right' ? null : lo, - hi: rule.tail === 'left' ? null : hi, - }; - } - - private reportBounds(variable: VariableRow): { lo: number | null; hi: number | null } | null { - const row = this.outlierPreviewRowsFor(variable)[0]; - if (!row) return null; - const lo = parseNumber(row.lowerBound); - const hi = parseNumber(row.upperBound); - if (lo === null && hi === null) return null; - return { lo, hi }; - } - - outlierBoundsView(variable: VariableRow): { - min: number; iqrLeft: number; iqrWidth: number; medianLeft: number; - loPct: number | null; hiPct: number | null; cappedLeft: number | null; cappedRight: number | null; - loLabel: string | null; hiLabel: string | null; note: string; - } | null { - const numbers = this.outlierDistributionNumbers(variable); - if (!numbers || numbers.max <= numbers.min) return null; - const span = numbers.max - numbers.min; - const pct = (value: number) => Math.max(0, Math.min(100, ((value - numbers.min) / span) * 100)); - const bounds = this.outlierBounds(variable); - const loPct = bounds?.lo != null ? pct(bounds.lo) : null; - const hiPct = bounds?.hi != null ? pct(bounds.hi) : null; - const reportRows = this.outlierPreviewRowsFor(variable); - const cappedCount = reportRows.reduce( - (sum, row) => sum + (parseNumber(row.lowerOutliers) ?? 0) + (parseNumber(row.upperOutliers) ?? 0), 0); - const note = bounds - ? `Below ${bounds.lo?.toLocaleString() ?? '—'} or above ${bounds.hi?.toLocaleString() ?? '—'} → capped` - + (reportRows.length && cappedCount > 0 ? ` · ${cappedCount.toLocaleString()} values` : '') - : 'Bounds shown after Refresh report'; - - return { - min: numbers.min, - iqrLeft: pct(numbers.q1), - iqrWidth: Math.max(2, pct(numbers.q3) - pct(numbers.q1)), - medianLeft: pct(numbers.median), - loPct, - hiPct, - cappedLeft: loPct, - cappedRight: hiPct !== null ? 100 - hiPct : null, - loLabel: bounds?.lo != null ? bounds.lo.toLocaleString() : null, - hiLabel: bounds?.hi != null ? bounds.hi.toLocaleString() : null, - note, - }; + outlierBoundsView(variable: VariableRow): OutlierBoundsView | null { + return outlierBoundsView( + outlierDistributionNumbers(this.distributionFor(variable)), + this.outlierRuleFor(variable), + this.outlierPreviewRowsFor(variable) + ); } outlierRuleFor(variable: VariableRow): OutlierRule { - if (!this.pendingOutlierRules[variable.code]) { - this.pendingOutlierRules = { - ...this.pendingOutlierRules, - [variable.code]: this.defaultOutlierRule(variable.code), - }; - } - return this.pendingOutlierRules[variable.code]; + return this.preprocessingState.outlierRule(variable.code); } onOutlierEnabledChange(variable: VariableRow, enabled: boolean): void { @@ -2948,7 +2177,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { private foldPreprocessingStation(): void { this.sectionOpen.update((open) => ({ ...open, setup: false, processed: false })); this.cdr.markForCheck(); - this.requestSectionScroll('setup'); + this.sectionAnchors.requestScroll('setup'); } /** @@ -2982,7 +2211,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { !!this.expStudioService.selectedDataModel() && this.expStudioService.selectedDatasets().length > 0; if (!hasScope || !variableCodes.length) { this.clearProcessedPreviewSubscription(); - this.processedSummary = this.createEmptySummary(false); + this.processedSummary = emptySummaryView(false); this.processedSummaryKey = ''; this.cdr.markForCheck(); return; @@ -2997,11 +2226,10 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { if (nextKey === this.processedSummaryKey) return; const previous = this.processedSummary; const requestSeq = ++this.processedDescribeSeq; - this.clearHistogramSubscriptions(this.processedSummary); this.clearProcessedPreviewSubscription(); this.processedSummaryKey = nextKey; this.processedSummary = { - ...this.createEmptySummary(true), + ...emptySummaryView(true), activeTab: previous.activeTab, selectedStatisticKey: previous.selectedStatisticKey, histogramSplitByVariable: previous.histogramSplitByVariable, @@ -3015,7 +2243,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { if (requestSeq !== this.processedDescribeSeq) return; this.clearProcessedPreviewSubscription(); this.processedSummary = this.buildSummaryFromResponse(response, 'processed'); - this.refreshActiveHistogramPreview('processed'); + this.summaryHistogram.refreshActive('processed'); this.emitProgressState(); this.cdr.markForCheck(); }, @@ -3026,7 +2254,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { // Drop the claimed key so reopening the preview retries instead of // trusting a summary that never arrived. this.processedSummaryKey = ''; - this.processedSummary = this.createEmptySummary(false); + this.processedSummary = emptySummaryView(false); this.cdr.markForCheck(); }, }); @@ -3052,7 +2280,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { : this.buildScopedPreprocessingConfig(scope); if (!preprocessing) { this.commitPendingToApplied(scope); - this.processedSummary = this.createEmptySummary(false); + this.processedSummary = emptySummaryView(false); this.persistAppliedDescriptivePreprocessing( this.buildPreprocessingConfig(this.appliedPreprocessingRules) ); @@ -3069,9 +2297,8 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const variableCodes = this.preprocessingVariables.map((variable) => variable.code); const requestSeq = ++this.processedDescribeSeq; this.isApplyingPreprocessing = true; - this.clearHistogramSubscriptions(this.processedSummary); this.clearProcessedPreviewSubscription(); - this.processedSummary = this.createEmptySummary(true); + this.processedSummary = emptySummaryView(true); // Claim the summary key up front so the navigate-to-preview fetch is // deduped against the apply request below (identical payload). this.processedSummaryKey = this.buildProcessedSummaryKey(variableCodes, preprocessing); @@ -3099,10 +2326,10 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.persistAppliedDescriptivePreprocessing( this.buildPreprocessingConfig(this.appliedPreprocessingRules) ); - this.userPreprocessingApplied = true; + this.preprocessingState.userApplied = true; this.preprocessingStatus = this.pendingChangeCount > 0 ? 'pending' : 'applied'; this.isApplyingPreprocessing = false; - this.refreshActiveHistogramPreview('processed'); + this.summaryHistogram.refreshActive('processed'); this.emitProgressState(); // Apply saves and closes: fold the card (or, for Apply & Continue, the station), // and do not open the next one. The summary just computed stays reachable @@ -3123,7 +2350,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { // Nothing was applied, so the key claimed for this request is a lie; // clear it so the next preview or apply refetches. this.processedSummaryKey = ''; - this.processedSummary = this.createEmptySummary(false); + this.processedSummary = emptySummaryView(false); if (scope === 'all') { this.sectionOpen.update((open) => ({ ...open, processed: true })); } else { @@ -3145,7 +2372,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.rawDescribeSeq++; this.rawSummaryKey = ''; this.expStudioService.clearDataExclusionWarnings(); - this.rawSummary = this.createEmptySummary(false); + this.rawSummary = emptySummaryView(false); this.processedData = []; this.isLoading = false; this.cdr.markForCheck(); @@ -3162,7 +2389,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.rawSummaryKey = ''; this.rawDescribeSeq++; this.expStudioService.clearDataExclusionWarnings(); - this.rawSummary = this.createEmptySummary(false); + this.rawSummary = emptySummaryView(false); this.processedData = []; this.isLoading = false; this.cdr.markForCheck(); @@ -3182,7 +2409,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.rawSummary = this.buildSummaryFromResponse(response, 'raw'); this.processedData = this.rawSummary.data; this.isLoading = false; - this.refreshActiveHistogramPreview('raw'); + this.summaryHistogram.refreshActive('raw'); this.cdr.markForCheck(); }, error: (err) => { @@ -3202,14 +2429,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { * states cannot reuse each other's response. */ private rawSummaryKeyFor(filterOverride?: BackendFilter | null): string { - const effectiveFilter = filterOverride === undefined - ? this.expStudioService.filterLogic() - : filterOverride; - return this.buildSelectionKey( - this.expStudioService.selectedVariables(), - this.expStudioService.selectedFilters(), - effectiveFilter, - ); + return this.statisticSummary.rawKeyFor(filterOverride); } /** @@ -3222,13 +2442,11 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } variableLabel(variable: VariableRow): string { - return variable.name ?? variable.label ?? variable.code; + return variableLabel(variable); } datasetLabel(datasetCode: string): string { - if (!datasetCode) return datasetCode; - const map = this.expStudioService.getDatasetLabelMap(); - return resolveDatasetDisplayLabel(String(datasetCode), map); + return datasetDisplayLabel(datasetCode, this.expStudioService.getDatasetLabelMap()); } variableTypeLabel(variable: VariableRow): string { @@ -3250,168 +2468,23 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { .filter((item): item is EnumOption => !!item); } - async exportAllDescriptiveToPDF(kind: SummaryKind = 'raw'): Promise { - this.isExporting = true; - this.cdr.markForCheck(); - try { - const numericCharts = document.querySelectorAll( - `.hidden-charts-for-export.${kind}-export .numeric-charts-export app-chart-renderer` - ) as NodeListOf; - - const nominalCharts = document.querySelectorAll( - `.hidden-charts-for-export.${kind}-export .nominal-charts-export app-chart-renderer` - ) as NodeListOf; - - const dataModel = this.expStudioService.selectedDataModel(); - const pathologyName = dataModel?.label || dataModel?.code || ''; - const summary = this.getSummary(kind); - - const labelMap = this.expStudioService.variableLabelMap(); - const enumMaps = this.expStudioService.getCategoricalEnumMaps(); - const filterLogic = this.expStudioService.filterLogic(); - await this.pdfExportService.exportDescriptiveStatisticsPdf({ - title: this.summaryKindLabel(kind), - pathologyName, - context: [ - ['Datasets', this.expStudioService.selectedDatasets().map((code) => this.datasetLabel(code))], - // Source data is the selection before any cohort rule or preprocessing. - ['Filters', kind !== 'source' && filterLogic ? formatFilterExpression(filterLogic, { labelMap, enumMaps }) : null], - ['Preprocessing', kind === 'processed' - ? this.expStudioService.formatPreprocessingConfig(this.expStudioService.getAppliedDescriptivePreprocessing(), labelMap, enumMaps).split('\n') - : null], - ], - variables: this.pivotBlocksWithDatasetLabels(summary.data), - charts: numericCharts, - nonNominalVariables: summary.nonNominalVariables, - nominalCharts, - nominalVariables: summary.nominalVariables, - mipVersion: this.mipVersion, - }); - } catch (err) { - console.error('PDF export failed:', err); - } finally { - this.isExporting = false; - this.cdr.markForCheck(); - } + exportAllDescriptiveToPDF(kind: SummaryKind = 'raw'): Promise { + return this.summaryExport.toPdf(kind, this.getSummary(kind)); } exportSummaryToCSV(kind: SummaryKind): void { - const summary = this.getSummary(kind); - if (!summary.data.length) return; - - const rows = summary.data.flatMap((variable) => - variable.rows.flatMap((row) => - variable.columns.map((dataset) => ({ - Variable: variable.name, - Metric: row.metric, - Dataset: this.datasetLabel(dataset), - Value: row.values[dataset] ?? '', - })) - ) - ); - - const filename = `${kind}_data_summary.csv`; - this.csvExportService.exportToCsv(rows, ['Variable', 'Metric', 'Dataset', 'Value'], filename); - } - - private createEmptySummary(isLoading: boolean): SummaryView { - return { - activeTab: 'Statistics', - data: [], - featurewiseRows: [], - isLoading, - showDistributions: false, - distributionSubTab: 'Numeric', - nonNominalVariables: [], - nominalVariables: [], - chartsForBoxPlot: [], - chartsForNominal: [], - histogramDataByVariable: {}, - histogramLoadingByVariable: {}, - histogramErrorByVariable: {}, - histogramSubscriptionsByVariable: {}, - histogramRequestIdByVariable: {}, - histogramSplitByVariable: {}, - activeBoxPlotIndex: 0, - activeNominalIndex: 0, - selectedStatisticKey: null, - }; + this.summaryExport.toCsv(kind, this.getSummary(kind)); } /** Every kind with a summary surface, in pipeline order. */ readonly summaryKinds: SummaryKind[] = ['source', 'raw', 'processed']; getSummary(kind: SummaryKind): SummaryView { - if (kind === 'raw') return this.rawSummary; - if (kind === 'source') return this.sourceSummary; - return this.processedSummary; - } - - private sectionElement(section: SectionKey): ElementRef | undefined { - switch (section) { - case 'source': - return this.sourceSection; - case 'raw': - return this.rawSection; - case 'setup': - return this.setupSection; - case 'filters': - return this.filtersSection; - case 'processed': - return this.processedSection; - case 'transformation': - return this.transformationSection; - default: { - const _exhaustive: never = section; - return _exhaustive; - } - } - } - - private scrollSectionIntoView(section: SectionKey): void { - const element = this.sectionElement(section)?.nativeElement; - if (!element) return; - - requestAnimationFrame(() => { - element.scrollIntoView({ behavior: 'smooth', block: 'start' }); - }); + return this.statisticSummary.get(kind); } private buildSummaryFromResponse(response: unknown, kind: SummaryKind): SummaryView { - const current = this.getSummary(kind); - const featurewise = getFeaturewiseDescribeRows(response); - const dsFromPayload = Array.from( - new Set((featurewise ?? []).map((item: any) => String(item.dataset))) - ) as string[]; - const datasetOrder = dsFromPayload - .filter((dataset) => dataset && dataset !== 'all datasets') - .concat('all datasets'); - const variableList = this.preprocessingVariables; - const data = this.pivotByDataset(featurewise, variableList, datasetOrder); - const distributionState = this.buildDistributionState(featurewise, response); - const summary: SummaryView = { - ...current, - data, - featurewiseRows: featurewise, - isLoading: false, - activeTab: current.activeTab, - selectedStatisticKey: this.nextStatisticSelection(data, current.selectedStatisticKey), - histogramDataByVariable: {}, - histogramLoadingByVariable: {}, - histogramErrorByVariable: {}, - histogramSubscriptionsByVariable: {}, - histogramRequestIdByVariable: {}, - ...distributionState, - }; - return summary; - } - - private refreshActiveHistogramPreview(kind: SummaryKind): void { - const summary = this.getSummary(kind); - if (summary.activeTab !== 'Histogram') return; - const block = this.selectedStatisticBlock(kind); - if (!block) return; - this.ensureHistogramForBlock(kind, block); + return this.statisticSummary.fromDescribeResponse(response, this.getSummary(kind), this.preprocessingVariables); } private buildProcessedSummaryKey( @@ -3425,155 +2498,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { }); } - private ensureHistogramForBlock(kind: SummaryKind, block: PivotBlock): void { - const code = this.variableCodeForBlock(block); - if (!code) return; - - const split = this.histogramSplitValue(kind, block); - const cacheKey = histogramCacheKey(code, split); - const summary = this.getSummary(kind); - if (summary.histogramSubscriptionsByVariable[cacheKey]) return; - if (summary.histogramLoadingByVariable[cacheKey]) return; - if (summary.histogramDataByVariable[cacheKey]) return; - - if (!split) { - const fromDescribe = this.buildHistogramFromDescribeCounts(kind, code, block); - if (fromDescribe) { - summary.histogramDataByVariable = { - ...summary.histogramDataByVariable, - [cacheKey]: fromDescribe, - }; - summary.histogramErrorByVariable = { - ...summary.histogramErrorByVariable, - [cacheKey]: '', - }; - this.cdr.markForCheck(); - return; - } - } - - summary.histogramLoadingByVariable = { - ...summary.histogramLoadingByVariable, - [cacheKey]: true, - }; - summary.histogramErrorByVariable = { - ...summary.histogramErrorByVariable, - [cacheKey]: '', - }; - this.cdr.markForCheck(); - - // `histogram_sql` cannot run without a missing-value strategy, so even the - // unprocessed surfaces send the default drop for the plotted variable and nothing - // else. The processed surface sends its pending rules — never the derived columns, - // which are this variable's output, not its input. Grouping variables are not given - // a drop strategy: that would remove rows from the overall total. - const preprocessingOverride = this.histogramPreprocessingFor(kind, code); - - // Step 0 previews the unfiltered cohort, so its histogram must drop the - // filter too - otherwise the tab would show the filtered cohort under a - // "no filters applied" heading. - const requestSeq = (summary.histogramRequestIdByVariable[cacheKey] ?? 0) + 1; - summary.histogramRequestIdByVariable = { - ...summary.histogramRequestIdByVariable, - [cacheKey]: requestSeq, - }; - // Step 0 describes the unfiltered source; a Cohort Filtering preview pins the raw - // histograms to the same unsaved cohort its table came from; the processed surface - // keeps the applied cohort (undefined = the stored rules). - const filterOverride = kind === 'source' ? null : kind === 'raw' ? this.rawPreviewFilter : undefined; - this.expStudioService - .getAlgorithmResults( - AlgorithmNames.HISTOGRAM, - [code], - null, - preprocessingOverride, - filterOverride, - split ? [split] : null, - ) - .pipe(takeUntil(this.destroyed)) - .subscribe({ - next: (response) => { - const nextSummary = this.getSummary(kind); - if (nextSummary.histogramRequestIdByVariable[cacheKey] !== requestSeq) return; - const parsed = this.parseHistogramResponse(response, code, block, kind, split); - if (parsed.data) { - nextSummary.histogramDataByVariable = { - ...nextSummary.histogramDataByVariable, - [cacheKey]: parsed.data, - }; - nextSummary.histogramErrorByVariable = { - ...nextSummary.histogramErrorByVariable, - [cacheKey]: '', - }; - } else { - nextSummary.histogramErrorByVariable = { - ...nextSummary.histogramErrorByVariable, - [cacheKey]: parsed.error, - }; - } - nextSummary.histogramLoadingByVariable = { - ...nextSummary.histogramLoadingByVariable, - [cacheKey]: false, - }; - nextSummary.histogramSubscriptionsByVariable = { - ...nextSummary.histogramSubscriptionsByVariable, - [cacheKey]: Subscription.EMPTY, - }; - this.cdr.markForCheck(); - }, - error: () => { - const nextSummary = this.getSummary(kind); - if (nextSummary.histogramRequestIdByVariable[cacheKey] !== requestSeq) return; - nextSummary.histogramLoadingByVariable = { - ...nextSummary.histogramLoadingByVariable, - [cacheKey]: false, - }; - nextSummary.histogramErrorByVariable = { - ...nextSummary.histogramErrorByVariable, - [cacheKey]: 'Failed to load histogram preview.', - }; - nextSummary.histogramSubscriptionsByVariable = { - ...nextSummary.histogramSubscriptionsByVariable, - [cacheKey]: Subscription.EMPTY, - }; - this.cdr.markForCheck(); - }, - }); - } - - /** - * What one numerical histogram is plotted under. Raw and step 0 only ever add the - * default drop-NaN for the plotted variable: neither surface may show preprocessing - * or transformations it does not describe. The processed surface follows the pending - * editor state, and a variable those rules leave alone still gets the default drop, - * because a histogram without a handler is a failed request, not an unhandled cohort. - */ - private histogramPreprocessingFor(kind: SummaryKind, code: string): PreprocessingConfig { - if (kind === 'processed') { - const pending = this.buildPreprocessingConfig( - this.pendingPreprocessingRules, - this.currentPreprocessingCodeSet(), - ); - return this.withDefaultDropFor(pending, code); - } - return this.withDefaultDropFor(null, code); - } - - /** Add `drop` for `code` unless the config already carries a strategy for it. */ - private withDefaultDropFor( - preprocessing: PreprocessingConfig | null, - code: string - ): PreprocessingConfig { - const config: PreprocessingConfig = { ...(preprocessing ?? {}) }; - const handler = { ...((config['missing_values_handler'] as Record) ?? {}) }; - const strategies = { ...((handler['strategies'] as Record) ?? {}) }; - if (strategies[code]) return config; - strategies[code] = 'drop'; - handler['strategies'] = strategies; - config['missing_values_handler'] = handler; - return config; - } - private clearProcessedPreviewSubscription(): void { this.processedPreviewSubscription?.unsubscribe(); this.processedPreviewSubscription = null; @@ -3584,128 +2508,12 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { this.sourcePreviewSubscription = null; } - private clearHistogramSubscriptions(summary: SummaryView): void { - Object.values(summary.histogramSubscriptionsByVariable).forEach((subscription) => subscription.unsubscribe()); - summary.histogramSubscriptionsByVariable = {}; - } - - private buildHistogramFromDescribeCounts( - kind: SummaryKind, - code: string, - block: PivotBlock - ): HistogramPreviewData | null { - const featurewise = this.getSummary(kind).featurewiseRows; - const row = featurewise.find( - (item: { variable?: string; dataset?: string }) => - item?.variable === code && String(item?.dataset ?? '') === 'all datasets' - ) ?? featurewise.find((item: { variable?: string }) => item?.variable === code); - const counts = (row as { data?: { counts?: Record } })?.data?.counts; - if (!counts || typeof counts !== 'object' || !Object.keys(counts).length) { - return null; - } - - const variable = this.statisticBlockVariable(block); - const binCodes = Object.keys(counts); - const binsWithLabels = this.mapBinsToEnumLabels(binCodes, variable?.enumerations); - const mappedCounts = binCodes.map((key) => { - const value = counts[key]; - return typeof value === 'number' && Number.isFinite(value) ? value : null; - }); - if (!mappedCounts.some((count) => count !== null && count > 0)) { - return null; - } - + /** The plotted variable as a histogram preview describes it. */ + private histogramVariable(kind: SummaryKind, block: PivotBlock): HistogramPreviewVariable { return { - bins: binsWithLabels.map((bin) => String(bin)), - counts: mappedCounts, - variableName: block.name, - variableType: this.statisticBlockType(kind, block), - }; - } - - private extractTransientHistogramError(response: unknown, payload: unknown): string | null { - const candidates: unknown[] = [payload, response]; - if (response && typeof response === 'object' && 'result' in (response as object)) { - candidates.push((response as { result?: unknown }).result); - } - - for (const candidate of candidates) { - if (typeof candidate === 'string' && candidate.trim()) { - return candidate.trim(); - } - if (!candidate || typeof candidate !== 'object' || Array.isArray(candidate)) { - continue; - } - const record = candidate as Record; - const data = record['data']; - if (typeof data === 'string' && data.trim()) { - return data.trim(); - } - const message = record['message']; - if (typeof message === 'string' && message.trim()) { - return message.trim(); - } - } - - return null; - } - - private parseHistogramResponse( - response: unknown, - code: string, - block: PivotBlock, - kind: SummaryKind, - groupingCode: string | null = null - ): { data: HistogramPreviewData | null; error: string } { - if (!response) { - return { data: null, error: 'Failed to load histogram preview.' }; - } - - const payload = (response as { result?: unknown })?.result ?? response; - const histList = extractHistogramItems(response); - const stacked = buildStackedHistogramSeries( - histList, - code, - groupingCode, - (groupingVar, groupingEnum) => this.groupingEnumLabel(groupingVar, groupingEnum), - this.expStudioService.selectedDatasets() - ); - if (!stacked) { - const backendError = this.extractTransientHistogramError(response, payload); - if (backendError) { - return { data: null, error: backendError }; - } - const fallback = kind === 'processed' - ? 'No histogram data available for this variable after preprocessing. The federated histogram preview returned no bins—often because preprocessing removed too many rows or the execution engine reported an error.' - : 'No histogram data available for this variable.'; - return { data: null, error: fallback }; - } - - const { bins, counts: numericCounts } = stacked; - if (!numericCounts.some((count) => count !== null && count > 0)) { - return { - data: null, - error: 'Histogram counts are unavailable (privacy threshold or insufficient data after preprocessing).', - }; - } - - const variable = this.statisticBlockVariable(block); - const binsWithLabels = this.mapBinsToEnumLabels(bins, variable?.enumerations); - const binLabels = binsWithLabels.map((bin) => String(bin)); - const splitLabel = groupingCode - ? (this.histogramSplitOptions(block).find((option) => option.code === groupingCode)?.label ?? groupingCode) - : undefined; - - return { - data: { - bins: binLabels, - counts: numericCounts, - groups: stacked.groups, - variableName: block.name, - variableType: this.statisticBlockType(kind, block), - splitLabel, - }, - error: '', + name: block.name, + type: this.statisticBlockType(kind, block), + enumerations: this.statisticBlockVariable(block)?.enumerations, }; } @@ -3715,69 +2523,11 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } const selected = (this.expStudioService.selectedVariables() as VariableRow[]) .find((variable) => variable.code === groupingVar); - const mapped = this.getEnumLabelMap(selected ?? this.findModelVariable(groupingVar) ?? undefined) + const mapped = enumLabelMap(selected ?? this.findModelVariable(groupingVar) ?? undefined) .get(groupingEnum); return mapped ?? groupingEnum; } - private buildDistributionState(featurewise: any[], response: unknown): Pick { - const selectedVars = this.expStudioService.selectedVariables() as VariableRow[]; - const unique = Array.from(new Map(selectedVars.map((v) => [v.code, v])).values()); - - const varsWithCounts = new Set(); - for (const item of featurewise) { - if (item.data && item.data.counts && Object.keys(item.data.counts).length > 0) { - varsWithCounts.add(item.variable); - } - } - - const nominalVariables = unique.filter( - (v) => v?.type === 'nominal' || varsWithCounts.has(v.code) - ); - const nominalCodes = new Set(nominalVariables.map((v) => v.code)); - const nonNominalVariables = unique.filter( - (v) => !nominalCodes.has(v.code) && v?.type !== 'text' - ); - const showDistributions = nonNominalVariables.length > 0 || nominalVariables.length > 0; - const distributionSubTab = nonNominalVariables.length > 0 ? 'Numeric' : 'Nominal'; - - const datasetLabels = this.expStudioService.getDatasetLabelMap(); - const chartsForBoxPlot = nonNominalVariables.map((v) => { - const perVarResp = { - ...(response as Record), - result: { - ...((response as { result?: Record })?.result ?? {}), - featurewise: featurewise.filter( - (row: any) => row.variable === v.code && row.dataset !== 'all datasets' - ), - dataset_labels: datasetLabels, - }, - }; - return this.chartBuilder.getChartsForAlgorithm('describe', perVarResp); - }); - - const chartsForNominal = nominalVariables.map((v) => { - const varData = featurewise.filter((row: any) => row.variable === v.code); - const varLabel = this.variableLabel(v); - const enumMap = this.getEnumLabelMap(v); - return buildGroupedBarChart(varData, varLabel, enumMap); - }); - - return { - showDistributions, - distributionSubTab, - nonNominalVariables, - nominalVariables, - chartsForBoxPlot, - chartsForNominal, - activeBoxPlotIndex: 0, - activeNominalIndex: 0, - }; - } - private buildOutlierPreviewRows(response: unknown): OutlierPreviewRow[] { const payload = (response as { result?: unknown })?.result ?? response; const featurewise = Array.isArray((payload as { featurewise?: unknown })?.featurewise) @@ -3798,13 +2548,13 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { dataset: String(item.dataset ?? '').trim() || '-', strategy: outlierStrategyLabel(String(data['strategy'] ?? '')), tail: outlierTailLabel(String(data['tail'] ?? '')), - fold: this.formatOutlierBound(data['fold']), - lowerBound: this.formatOutlierBound(data['lower_bound']), - upperBound: this.formatOutlierBound(data['upper_bound']), - lowerOutliers: this.formatInteger(data['lower_outlier_count'], '-'), - upperOutliers: this.formatInteger(data['upper_outlier_count'], '-'), - totalOutliers: this.formatInteger(data['total_outlier_count'], '-'), - outlierPercentage: this.formatOutlierPercentage(data['total_outlier_percentage']), + fold: formatOutlierBound(data['fold']), + lowerBound: formatOutlierBound(data['lower_bound']), + upperBound: formatOutlierBound(data['upper_bound']), + lowerOutliers: formatInteger(data['lower_outlier_count'], '-'), + upperOutliers: formatInteger(data['upper_outlier_count'], '-'), + totalOutliers: formatInteger(data['total_outlier_count'], '-'), + outlierPercentage: formatOutlierPercentage(data['total_outlier_percentage']), }; }); } @@ -3814,40 +2564,10 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return variable ? this.variableLabel(variable) : code; } - /** Canonical (painted + exported) pivot cell: fixed 2 decimals, no grouping. */ - private formatNumber(value: unknown): string { - if (value === null || value === undefined || value === '') return 'N/A'; - const n = typeof value === 'number' ? value : Number(String(value).replace(/,/g, '')); - if (!Number.isFinite(n)) return String(value); - return n.toFixed(2); - } - /** Outlier preview bounds keep the old 6-decimal preview precision. */ - private formatOutlierBound(value: unknown): string { - if (value === null || value === undefined || value === '') return '-'; - const n = typeof value === 'number' ? value : Number(String(value).replace(/,/g, '')); - if (!Number.isFinite(n)) return String(value); - return n.toLocaleString(undefined, { maximumFractionDigits: 6 }); - } - private formatInteger(value: unknown, empty = 'N/A'): string { - if (value === null || value === undefined || value === '') return empty; - const n = typeof value === 'number' ? value : Number(value); - if (!Number.isFinite(n)) return String(value); - return String(Math.round(n)); - } - private formatPercent(pct: number, digits?: number): string { - const d = digits ?? (pct >= 10 || pct === 0 ? 0 : 1); - return `${pct.toFixed(d)}%`; - } - private formatOutlierPercentage(value: unknown): string { - if (value === null || value === undefined || value === '') return '-'; - const num = typeof value === 'number' ? value : parseFloat(String(value)); - if (!Number.isFinite(num)) return '-'; - return this.formatPercent(num, 2); - } /** Copy pending → applied for one card or all of them. */ private commitPendingToApplied(scope: PrepStep | 'all'): void { @@ -3891,322 +2611,23 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return Object.keys(config).length ? config : null; } - /** Build the exaflow config for one card, or null when it is not yet complete. */ - buildTransformationConfigForDraft(draft: TransformationColumnDraft): Record | null { - const code = draft.code.trim(); - if (!code) return null; - const rules: Record = {}; - for (const r of draft.rules) { - const value = r.value.trim(); - if (value && r.filter) rules[value] = r.filter; - } - if (!Object.keys(rules).length) return null; - // Exaflow names the created column with `new_column_names`; the draft keeps `code` as the - // editable UI name only. The emitted creator parameters never carry a `code` key. - const config: Record = { - new_column_names: [code], - strategy: 'filter_rules', - rules, - }; - const def = draft.defaultEnumeration.trim(); - if (def) config['default_enumeration'] = def; - return config; - } - onTransformationChange(): void { - // Each categorical draft upserts by id. An incomplete draft clears its outputs so the - // request skips it. A draft with a write problem leaves the prior parameters untouched and + // Each nominal draft upserts by id. An incomplete draft clears its outputs so the request + // skips it. A draft with a write problem leaves the prior parameters untouched and // shows the problem on its status chip. for (const draft of this.transformationDrafts) { - if (this.draftWriteProblem(draft)) continue; - const config = this.buildTransformationConfigForDraft(draft); + if (this.transformationDraftState.writeProblem(draft)) continue; this.expStudioService.upsertTransformation({ id: draft.id, name: 'categorical_column_creator', - parameters: config ?? {}, + parameters: transformationConfigFor(draft) ?? {}, }); } - this.invalidateCategoryCounts(); + this.transformationStats.invalidate(); this.emitProgressState(); this.cdr.markForCheck(); } - /** Applied K-means creators from store cards (every card with an output name). */ - private appliedKMeansCreators(): Array<{ code: string; creator: Record }> { - return this.cards() - .filter((step) => step.name === 'kmeans_cluster_creator') - .map((step) => { - const code = String((step.parameters['new_column_names'] as unknown[] | undefined)?.[0] ?? '').trim(); - return { code, creator: step.parameters }; - }) - .filter((entry) => !!entry.code); - } - - /** Category counts describe the config at the last Done; any later edit drops them. */ - private invalidateCategoryCounts(): void { - if (this.countsDraftId === null) return; - this.countsDraftId = null; - // ponytail: also cancels an in-flight K-means preview; scope per card if both overlap in practice. - this.transformationStatsRequestId++; - this.isTransformationStatsLoading = false; - this.transformationStatisticsError = ''; - const clusterCodes = new Set(this.appliedKMeansCreators().map((entry) => entry.code)); - this.transformationStatistics = this.transformationStatistics.filter((block) => clusterCodes.has(block.code)); - } - - /** Placeholder (null-count) rows for one card so Preview shows the category shape. */ - private placeholderRowsForDraft(draft: TransformationColumnDraft): Array<{ value: string; count: number | null }> { - const configuredValues = draft.rules.map((r) => r.value.trim()).filter((v) => v.length > 0); - const rows: Array<{ value: string; count: number | null }> = configuredValues.map((value) => ({ value, count: null })); - const def = draft.defaultEnumeration.trim(); - if (def && !configuredValues.includes(def)) { - rows.push({ value: def, count: null }); - } - return rows; - } - - /** - * Why no card reached a complete config, phrased once for every named card that is short - * of one: a card with no named category needs a category, a card whose categories are - * unnamed by a filter needs the filter. A stage with no named cards at all has nothing to - * explain — the empty state already says so. - */ - private transformationStatsGapNotice(): string { - const gaps: string[] = []; - if (this.transformationDrafts.some((draft) => draft.code.trim() - && !draft.rules.some((rule) => rule.value.trim().length > 0))) { - gaps.push('a named category'); - } - if (this.transformationRulesNeedFilters) { - gaps.push('a filter on each one'); - } - return gaps.length - ? `Each derived column needs ${gaps.join(' and ')} before counts can be loaded.` - : ''; - } - - /** - * The counts note belongs to rule-built columns. A K-means column carries its own - * sentence, in the same shape as the categorical column card. - */ - showsRuleColumnCounts(): boolean { - const clusterCodes = new Set(this.appliedKMeansCreators().map((entry) => entry.code)); - return this.transformationStatistics.some((block) => !clusterCodes.has(block.code)); - } - - /** - * Present an applied K-means column as a categorical column: a name, one assignment - * rule per cluster, and the variables that placed each record. Null for any other column. - */ - kmeansColumnView(code: string): { - code: string; - lead: string; - foot: string; - rules: Record; - } | null { - const target = code.trim(); - const match = this.appliedKMeansCreators().find((entry) => entry.code === target); - if (!match) return null; - - const sources = kmeansCreatorVariables(match.creator); - const variables = sources - .map((variable) => this.variableLabelForCode(variable)) - .filter((label, index, all) => !!label && label !== sources[index] && all.indexOf(label) === index); - const choices = kmeansCreatorClusterChoices(match.creator); - const rules: Record = {}; - for (const choice of choices) { - const label = choice.label.trim(); - rules[choice.cluster_id] = label && label !== choice.cluster_id - ? `Nearest cluster — ${label}` - : 'Nearest cluster'; - } - const count = choices.length; - const clusterWord = count === 1 ? 'cluster' : 'clusters'; - return { - code: target, - lead: variables.length - ? `Each record is assigned to the nearest of these clusters, using ${variables.join(', ')}.` - : 'Each record is assigned to the nearest of these clusters.', - foot: variables.length - ? `${count} ${clusterWord} from ${variables.join(', ')} · covers every record that has those variables` - : `${count} ${clusterWord} · covers every record`, - rules, - }; - } - - refreshTransformationStatistics(): void { - // One stats table per derived column (categorical cards and every applied K-means - // cluster column); counts come from a single describe over all of them. - const kmeansEntries = this.appliedKMeansCreators(); - const kmeansPlaceholders = kmeansEntries.map((entry) => { - const values = kmeansCreatorClusterChoices(entry.creator).map((choice) => choice.cluster_id); - return { code: entry.code, rows: values.map((value) => ({ value, count: null as number | null })) }; - }); - const placeholderBlocks = [ - ...this.transformationDrafts - .filter((draft) => draft.code.trim().length > 0) - .map((draft) => ({ code: draft.code.trim(), rows: this.placeholderRowsForDraft(draft) })), - ...kmeansPlaceholders, - ]; - const configs = this.transformationConfigs(); - - if (!configs.length && !kmeansEntries.length) { - this.transformationStatistics = placeholderBlocks; - // Every named card gets a heading over this table, so a named card always needs a - // reason beside it: falling through to the "counts from a describe run" line would - // describe a describe that never ran. - this.transformationStatisticsError = this.transformationStatsGapNotice(); - this.isTransformationStatsLoading = false; - this.cdr.markForCheck(); - return; - } - - const columnCodes = [ - ...configs.map((config) => String((config['new_column_names'] as unknown[] | undefined)?.[0] ?? '').trim()).filter(Boolean), - ...kmeansEntries.map((entry) => entry.code), - ]; - const columnSet = new Set(columnCodes); - // Same CDE set a run would send: filter fields + selected variables + K-means/PCA sources - // that are not earlier outputs. Created columns (pca_pc*, cluster codes) stay out. - const sourceCodes = Array.from( - new Set( - [ - ...configs.flatMap((config) => - Object.values((config['rules'] as Record) ?? {}).flatMap((filter) => - this.expStudioService.filterVariableCodes(filter) - ) - ), - ...this.expStudioService.transformationSourceCodesForInput(), - ...this.expStudioService.selectedVariables().map((v) => String(v?.code ?? '')), - ] - .map((code) => String(code).trim()) - .filter((code) => !!code && !columnSet.has(code)) - ) - ); - - if (!sourceCodes.length) { - this.transformationStatistics = placeholderBlocks; - this.transformationStatisticsError = configs.length - ? 'Category filters must reference at least one data-model variable before counts can be loaded.' - : 'The K-means cluster column has no clustering variables to read.'; - this.isTransformationStatsLoading = false; - this.cdr.markForCheck(); - return; - } - - const requestId = ++this.transformationStatsRequestId; - this.isTransformationStatsLoading = true; - this.transformationStatisticsError = ''; - this.transformationStatistics = placeholderBlocks; - this.cdr.markForCheck(); - - // Pipeline consistency: the preview uses the same ordered-step builder as a run - // (singletons, then PCA/categorical/K-means in store order with CDE-only drops). - if (Object.keys(this.validatePendingRules()).length > 0) { - this.transformationStatistics = placeholderBlocks; - this.transformationStatisticsError = 'Fix the preprocessing rules before loading category counts.'; - this.isTransformationStatsLoading = false; - this.cdr.markForCheck(); - return; - } - const preprocessing = this.expStudioService.buildTransformationDescribePreprocessing( - this.buildPreprocessingConfig(this.pendingPreprocessingRules), - ); - this.expStudioService - .loadDescriptiveOverview(columnCodes, preprocessing, sourceCodes) - .subscribe({ - next: (response) => { - if (requestId !== this.transformationStatsRequestId) return; - if (!response) { - this.transformationStatisticsError = 'Failed to load category counts.'; - this.isTransformationStatsLoading = false; - this.cdr.markForCheck(); - return; - } - this.transformationStatistics = [ - ...configs.map((config) => { - const code = String((config['new_column_names'] as unknown[] | undefined)?.[0] ?? '').trim(); - const rules = (config['rules'] as Record) ?? {}; - const def = String(config['default_enumeration'] ?? '').trim(); - return { - code, - rows: this.buildTransformationStatisticsFromDescribe(response, code, Object.keys(rules), def), - }; - }), - ...kmeansEntries.map((entry) => { - const values = kmeansCreatorClusterChoices(entry.creator).map((choice) => choice.cluster_id); - return { - code: entry.code, - rows: this.buildTransformationStatisticsFromDescribe(response, entry.code, values, ''), - }; - }), - ]; - const hasAnyCount = this.transformationStatistics.some((block) => - block.rows.some((row) => typeof row.count === 'number' && Number.isFinite(row.count)) - ); - this.transformationStatisticsError = hasAnyCount - ? '' - : this.isErrorResponse(response) - ? 'Failed to load category counts.' - : 'No counts were returned (privacy threshold or insufficient data after preprocessing).'; - this.isTransformationStatsLoading = false; - this.cdr.markForCheck(); - }, - error: () => { - if (requestId !== this.transformationStatsRequestId) return; - this.transformationStatisticsError = 'Failed to load category counts.'; - this.isTransformationStatsLoading = false; - this.cdr.markForCheck(); - }, - }); - } - - private isErrorResponse(response: unknown): boolean { - const r = response as { status?: unknown; result?: { type?: unknown; data?: unknown } } | null; - return r?.status === 'error' || - (typeof r?.result?.type === 'string' && r.result.type.includes('error')) || - typeof r?.result?.data === 'string'; - } - - /** Category counts as the same horizontal bar chart the summary uses for groups. */ - transformationCountChart(block: { code: string; rows: Array<{ value: string; count: number | null }> }): HistogramPreviewData | null { - if (!block.rows.some((row) => typeof row.count === 'number')) return null; - return { - bins: block.rows.map((row) => row.value), - counts: block.rows.map((row) => row.count), - variableName: block.code, - variableType: 'nominal', - }; - } - - private buildTransformationStatisticsFromDescribe( - response: unknown, - columnCode: string, - configuredValues: string[], - defaultEnumeration: string - ): Array<{ value: string; count: number | null }> { - const featurewise = getFeaturewiseDescribeRows(response); - const row = - featurewise.find( - (item: { variable?: string; dataset?: string }) => - item?.variable === columnCode && String(item?.dataset ?? '') === 'all datasets' - ) ?? - featurewise.find((item: { variable?: string }) => item?.variable === columnCode); - const counts = (row as { data?: { counts?: Record } })?.data?.counts ?? {}; - const ordered = [...configuredValues]; - if (defaultEnumeration && !ordered.includes(defaultEnumeration)) { - ordered.push(defaultEnumeration); - } - Object.keys(counts).forEach((key) => { - if (!ordered.includes(key)) ordered.push(key); - }); - return ordered.map((value) => { - const raw = counts[value]; - const count = typeof raw === 'number' && Number.isFinite(raw) ? raw : null; - return { value, count }; - }); - } - private buildMissingPreprocessingConfig( rules: Record, allowedCodes = this.currentPreprocessingCodeSet() @@ -4360,13 +2781,13 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { // default). Create mode must stay out: auto-applied defaults there are // persisted into the applied config without a user apply. if (this.expStudioService.editingExistingExperiment()) { - this.userPreprocessingApplied = true; + this.preprocessingState.userApplied = true; } } else { this.ensureDefaultRulesForCurrentSelection(currentCodes); this.ensureOutlierDefaultsForCurrentSelection(); // No persisted config means this session has no explicit apply yet. - this.userPreprocessingApplied = false; + this.preprocessingState.userApplied = false; } this.preprocessingValidationErrors = Object.fromEntries( Object.entries(this.preprocessingValidationErrors).filter(([code]) => currentCodes.has(code)) @@ -4395,14 +2816,14 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } | undefined; const strategies = missingValues?.strategies ?? {}; const fillValues = missingValues?.fill_values ?? {}; - const nextApplied = this.cloneRules(this.appliedPreprocessingRules); - const nextPending = this.cloneRules(this.pendingPreprocessingRules); + const nextApplied = cloneMissingRules(this.appliedPreprocessingRules); + const nextPending = cloneMissingRules(this.pendingPreprocessingRules); currentCodes.forEach((code) => { const strategy = String(strategies[code] ?? ''); const action = this.isMissingAction(strategy) ? strategy : 'no_action'; const rule: PreprocessingRule = action === 'no_action' - ? this.emptyRule(code) + ? emptyMissingRule(code) : { variableCode: code, action, @@ -4413,9 +2834,9 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const existingPending = nextPending[code]; nextPending[code] = action === 'no_action' ? { - ...(existingPending && this.serializeRule(existingPending) !== this.serializeRule(rule) + ...(existingPending && serializeMissingRule(existingPending) !== serializeMissingRule(rule) ? existingPending - : this.defaultRule(code)), + : defaultMissingRule(code)), } : { ...rule }; }); @@ -4512,11 +2933,11 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } private ensureDefaultRulesForCurrentSelection(currentCodes = this.currentPreprocessingCodeSet()): void { - const nextPending = this.cloneRules(this.pendingPreprocessingRules); + const nextPending = cloneMissingRules(this.pendingPreprocessingRules); let changed = false; currentCodes.forEach((code) => { if (!nextPending[code] && !this.appliedPreprocessingRules[code]) { - nextPending[code] = this.defaultRule(code); + nextPending[code] = defaultMissingRule(code); changed = true; } }); @@ -4542,7 +2963,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { // hydrated defaults stay out of the emitted status so the studio stepper // does not mark Data review Done without user work. preprocessingStatus: - this.preprocessingStatus === 'applied' && !this.userPreprocessingApplied + this.preprocessingStatus === 'applied' && !this.preprocessingState.userApplied ? 'none' : this.preprocessingStatus, transformationStatusLabel: this.transformationStatusLabel, @@ -4631,8 +3052,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } private hasAppliedPreprocessing(variableCode: string): boolean { - const rule = this.appliedPreprocessingRules[variableCode]; - return !!rule?.enabled && rule.action !== 'no_action'; + return this.preprocessingState.hasAppliedMissing(variableCode); } private hasAppliedLongitudinalPreprocessing(variableCode: string): boolean { @@ -4642,16 +3062,16 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } private hasAppliedOutlierPreprocessing(variableCode: string): boolean { - return !!this.appliedOutlierRules[variableCode]?.enabled; + return this.preprocessingState.hasAppliedOutlier(variableCode); } private mergeRulesForCurrentSelection( base: Record, source: Record ): Record { - const next = this.cloneRules(base); + const next = cloneMissingRules(base); this.currentPreprocessingCodeSet().forEach((code) => { - next[code] = source[code] ? { ...source[code] } : this.defaultRule(code); + next[code] = source[code] ? { ...source[code] } : defaultMissingRule(code); }); return next; } @@ -4667,67 +3087,20 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return next; } - // For Missing Values the default action is `drop` (NA removal): rows with - // missing values are removed. Adding a variable does not create a pending - // change — the default is implicit, and the user can opt into a different - // imputation strategy (mean, median, constant) when needed. - private defaultRule(variableCode: string): PreprocessingRule { - return { variableCode, action: 'drop', value: '', enabled: true }; - } - private defaultOutlierRule(variableCode: string): OutlierRule { return createDefaultOutlierRule(variableCode, false); } - private emptyRule(variableCode: string): PreprocessingRule { - return { variableCode, action: 'no_action', value: '', enabled: false }; - } - - private cloneRules(rules: Record): Record { - return Object.fromEntries( - Object.entries(rules).map(([code, rule]) => [code, { ...rule }]) - ); - } - - private serializeRule(rule: PreprocessingRule | undefined): string { - const normalized = rule ?? this.emptyRule(''); - return JSON.stringify({ - action: normalized.action, - value: normalized.value, - enabled: normalized.enabled, - }); - } - - private buildSelectionKey( - variables: VariableRow[], - filters: VariableRow[], - filterLogic: unknown, - // Raw's own dependency set omits applied preprocessing on purpose: the raw describe - // is requested with a null config, so applying preprocessing cannot change its numbers. - appliedPreprocessing?: unknown - ): string { - return JSON.stringify({ - variables: variables.map((variable) => variable.code).sort(), - filters: filters.map((variable) => variable.code).sort(), - filterLogic, - appliedPreprocessing, - }); - } - private discardPendingTransformation(): void { this.hydrateAppliedTransformation(); if (!this.transformationDrafts.length) { - this.transformationStatistics = []; - this.transformationStatisticsError = ''; + this.transformationStats.clear(); } this.onTransformationChange(); } private getSummaryTotalRows(summary: SummaryView): number | null { - const all = (summary.featurewiseRows ?? []).find((item: any) => item.dataset === 'all datasets'); - if (!all) return null; - const total = Number(all.data?.num_total); - return Number.isFinite(total) ? total : null; + return pooledTotalRows(summary.featurewiseRows ?? []); } get cohortRetention(): { raw: number; processed: number; dropped: number; percentage: number } | null { @@ -4755,7 +3128,7 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { } isCategoricalVariable(variable: VariableRow): boolean { - return String(variable.type ?? '').toLowerCase() === 'nominal' || !!variable.enumerations?.length; + return isCategoricalVariable(variable); } private enumCodes(variable: VariableRow): string[] { @@ -4795,145 +3168,6 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { return this.variableForBlock(block)?.code ?? null; } - private mapBinsToEnumLabels( - bins: unknown[], - enumerations?: Array<{ code?: unknown; label?: string; name?: string }> - ): unknown[] { - if (!Array.isArray(bins) || !enumerations?.length) return bins; - - const codeToLabel = new Map( - enumerations.map((entry) => [ - String(entry?.code ?? entry?.label ?? entry?.name ?? ''), - entry?.label ?? entry?.name ?? String(entry?.code ?? ''), - ]) - ); - - let mapped = 0; - const mappedBins = bins.map((bin) => { - const label = codeToLabel.get(String(bin)); - if (label !== undefined) { - mapped += 1; - return label; - } - return bin; - }); - - return mapped > 0 ? mappedBins : bins; - } - - private getEnumLabelMap(variable: VariableRow | undefined): Map { - const enums = Array.isArray(variable?.enumerations) ? variable.enumerations : []; - const map = new Map(); - enums.forEach((item) => { - const raw = item?.code ?? item?.label ?? item?.name; - if (raw === null || raw === undefined) return; - const key = String(raw); - const label = String(item?.label ?? item?.name ?? raw); - map.set(key, label); - }); - return map; - } - - private pivotBlocksWithDatasetLabels(blocks: PivotBlock[]): PivotBlock[] { - return blocks.map((block) => { - const labeledColumns = block.columns.map((code) => this.datasetLabel(code)); - return { - ...block, - columns: labeledColumns, - rows: block.rows.map((row) => ({ - metric: row.metric, - values: Object.fromEntries( - block.columns.map((code, index) => [labeledColumns[index], row.values[code]]) - ), - })), - }; - }); - } - - private pivotByDataset( - items: any[], - variableList: VariableRow[], - datasetOrder: string[] - ): PivotBlock[] { - const byVar: Record = {}; - for (const item of items || []) { - const arr = byVar[item.variable] || (byVar[item.variable] = []); - arr.push(item); - } - - const result: PivotBlock[] = []; - for (const [varCode, arr] of Object.entries(byVar)) { - const matched = variableList.find((v) => v.code === varCode); - const varName = matched ? this.variableLabel(matched) : varCode; - const enumMap = this.getEnumLabelMap(matched); - const byDataset: Record = {}; - for (const entry of arr) byDataset[entry.dataset] = entry.data || {}; - - const countsKeys = new Set(); - for (const ds of datasetOrder) { - const counts = byDataset[ds]?.counts ?? {}; - Object.keys(counts).forEach((key) => countsKeys.add(String(key))); - } - - let rows: Array<{ metric: string; values: Record }> = []; - if (countsKeys.size > 0) { - const baseMetrics = [ - { key: 'num_datapoints', label: 'Datapoints' }, - { key: 'num_missing', label: 'Missing' }, - { key: 'num_total', label: 'Total' }, - ]; - rows = baseMetrics.map((metric) => { - const values: Record = {}; - for (const ds of datasetOrder) { - const raw = - metric.key === 'num_datapoints' ? byDataset[ds]?.num_dtps : - metric.key === 'num_missing' ? byDataset[ds]?.num_na : - byDataset[ds]?.num_total; - values[ds] = this.formatInteger(raw); - } - return { metric: metric.label, values }; - }); - - const orderedKeys = Array.from(enumMap.keys()).filter((key) => countsKeys.has(key)); - const remainingKeys = Array.from(countsKeys).filter((key) => !enumMap.has(key)); - orderedKeys.concat(remainingKeys).forEach((key) => { - const label = enumMap.get(key) ?? key; - const values: Record = {}; - for (const ds of datasetOrder) { - values[ds] = this.formatInteger(byDataset[ds]?.counts?.[key]); - } - rows.push({ metric: label, values }); - }); - } else { - const isCategorical = !!matched && this.isCategoricalVariable(matched); - const countKeys = new Set(['num_datapoints', 'num_missing', 'num_total']); - const metrics = isCategorical - ? this.metricOrder.filter((metric) => countKeys.has(metric.key)) - : this.metricOrder; - rows = metrics.map((metric) => { - const values: Record = {}; - for (const ds of datasetOrder) { - const raw = - metric.key === 'num_datapoints' ? byDataset[ds]?.num_dtps : - metric.key === 'num_missing' ? byDataset[ds]?.num_na : - metric.key === 'num_total' ? byDataset[ds]?.num_total : - metric.key === 'std' ? byDataset[ds]?.std : - metric.key === 'q2' ? byDataset[ds]?.q2 : - byDataset[ds]?.[metric.key]; - values[ds] = countKeys.has(metric.key) ? this.formatInteger(raw) : this.formatNumber(raw); - } - return { metric: metric.label, values }; - }); - } - result.push({ code: varCode, name: varName, columns: datasetOrder, rows }); - } - return result; - } - - private statisticBlockKey(block: PivotBlock): string { - return block.code ?? block.name; - } - private statisticBlockType(kind: SummaryKind, block: PivotBlock): StatisticVariableType { const summary = this.getSummary(kind); const variable = this.statisticBlockVariable(block); @@ -4942,27 +3176,4 @@ export class StatisticAnalysisPanelComponent implements OnDestroy { const numericMetrics = new Set(['Mean', 'Standard Deviation', 'Minimum', 'Q1', 'Median', 'Q3', 'Maximum']); return block.rows.some((row) => numericMetrics.has(row.metric)) ? 'numeric' : 'nominal'; } - - private statisticMetricText(block: PivotBlock, metric: string): string { - const row = block.rows.find((item) => item.metric === metric); - if (!row) return 'N/A'; - const dataset = block.columns.includes('all datasets') ? 'all datasets' : block.columns[0]; - return row.values[dataset] ?? 'N/A'; - } - - private statisticMetricNumber(block: PivotBlock, metric: string): number { - const value = Number(this.statisticMetricText(block, metric).replace(/,/g, '')); - return Number.isFinite(value) ? value : 0; - } - - private statisticMissingValue(block: PivotBlock): number { - return Math.round(this.statisticMetricNumber(block, 'Missing')); - } - - private nextStatisticSelection(data: PivotBlock[], currentKey: string | null): string | null { - if (!data.length) return null; - if (currentKey && data.some((block) => this.statisticBlockKey(block) === currentKey)) return currentKey; - return this.statisticBlockKey(data[0]); - } - } diff --git a/src/app/pages/experiment-studio/data-handling/data-handling.context.ts b/src/app/pages/experiment-studio/data-handling/data-handling.context.ts new file mode 100644 index 000000000..2f6c3d884 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/data-handling.context.ts @@ -0,0 +1,13 @@ +import { InjectionToken } from '@angular/core'; +import type { DataHandlingComponent } from './data-handling.component'; + +/** + * The Data Handling component, as seen by the sections it renders. + * + * Section components are views over the pipeline's state, which stays on + * DataHandlingComponent (its public API, the studio's calls into it and the specs all + * go through it). Sections inject this token instead of taking dozens of inputs, and + * use ChangeDetectionStrategy.Eager so they refresh whenever the pipeline does, + * exactly as the inline template did. + */ +export const DATA_HANDLING = new InjectionToken('DATA_HANDLING'); diff --git a/src/app/pages/experiment-studio/data-handling/data-handling.models.ts b/src/app/pages/experiment-studio/data-handling/data-handling.models.ts new file mode 100644 index 000000000..427982b35 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/data-handling.models.ts @@ -0,0 +1,168 @@ +import { EChartsOption } from 'echarts'; +import { VariableStats } from '../../../models/algorithm-results.model'; +import { BackendFilter, BackendRule } from '../../../models/filters.model'; +import { HistogramGroupSeries } from '../visualisations/histogram/histogram-data'; + +/** Shared types of the Data Handling pipeline and its subcomponents. */ + +export type TabKey = 'Statistics' | 'Charts' | 'Histogram'; +export type SummaryKind = 'source' | 'raw' | 'processed'; +export type SectionKey = 'source' | 'raw' | 'setup' | 'filters' | 'processed' | 'transformation'; +export type PipelineStep = 'source' | 'filters' | 'setup' | 'transformation'; +export type PrepStep = 'missing' | 'outlier' | 'longitudinal'; +export type DistributionSubTab = 'Numeric' | 'Nominal'; +export type StatisticVariableType = 'numeric' | 'nominal'; +export type PreprocessingStatus = 'none' | 'pending' | 'applied'; +export type PrepKind = 'missing' | 'outlier' | 'longitudinal'; + +export type SummaryAnchorKey = 'charts' | 'details' | 'export'; + +/** + * Where each summary surface wears a `data-guide` anchor. They are not uniform: the Raw + * surface owns the overlay-tab and export anchors, the Processed surface only its details + * panel, and step 0 is a read-only duplicate of the Raw surface, so it wears none — the + * same anchor on a hidden node would hand the tour or a QA selector the wrong surface. + */ +export const SUMMARY_GUIDE_ANCHORS: Record> = { + source: { charts: null, details: null, export: null }, + raw: { charts: 'analysis-charts', details: 'analysis-raw-details', export: 'analysis-export' }, + processed: { charts: null, details: 'analysis-processed-details', export: null }, +}; + +export type MissingAction = 'no_action' | 'drop' | 'mean' | 'median' | 'constant'; +export type LongitudinalStrategy = 'first' | 'second' | 'diff'; + +/** + * The record-level rule behind one category. exaflow takes either a group of rules or a + * single condition, and saved experiments hold both shapes - so the draft keeps whichever + * it was given instead of pretending a bare condition cannot exist. + */ +export type CategoryFilter = BackendFilter | BackendRule; + +export interface TransformationRule { + value: string; + filter: CategoryFilter | null; +} + +/** One "Create new categorical column" card. The stage holds an ordered list. */ +export interface TransformationColumnDraft { + /** Matches the store step id, so the drafts and the ordered cards stay aligned. */ + id: string; + code: string; + defaultEnumeration: string; + rules: TransformationRule[]; + open: boolean; + /** Rule index whose filter editor is expanded; at most one open per card. */ + openRuleIndex?: number | null; + /** Picked from the type chooser (or restored from the store). */ + chosen?: boolean; +} + +/** One sub-step of a pipeline stage, rendered on the pipeline-overview rail. */ +export interface PipelineSubNode { + // Preprocessing stations use fixed ids; transformation sub-nodes carry a + // `transformation:` id so the rail can list one card per derived column. + id: string; + icon: string; + title: string; + subtitle: string; + statusLabel: string; + statusTone: 'applied' | 'default' | 'pending'; + /** Optional consequence line (e.g. rows the default drop-NaN would remove). */ + impact?: string; +} + +export type MetricKey = + | 'num_dtps' | 'num_na' | 'num_total' + | 'mean' | 'std' | 'min' | 'q1' | 'q2' | 'q3' | 'max'; + +export interface SummaryView { + activeTab: TabKey; + data: PivotBlock[]; + /** The describe payload's featurewise rows behind `data` (one per variable x dataset). */ + featurewiseRows: VariableStats[]; + isLoading: boolean; + showDistributions: boolean; + distributionSubTab: DistributionSubTab; + nonNominalVariables: VariableRow[]; + nominalVariables: VariableRow[]; + chartsForBoxPlot: EChartsOption[][]; + chartsForNominal: EChartsOption[][]; + histogramDataByVariable: Record; + histogramLoadingByVariable: Record; + histogramErrorByVariable: Record; + /** Cache keys whose request has answered (data or error); not repeated until the surface is rebuilt. */ + histogramSettledByVariable: Record; + histogramRequestIdByVariable: Record; + histogramSplitByVariable: Record; + activeBoxPlotIndex: number; + activeNominalIndex: number; + selectedStatisticKey: string | null; +} + +export interface PivotBlock { + code?: string; + name: string; + columns: string[]; + rows: Array<{ metric: string; values: Record }>; +} + +export interface VariableRow { + code: string; + name?: string; + label?: string; + type?: string; + enumerations?: Array<{ code?: string; label?: string; name?: string }>; +} + +export interface EnumOption { + code: string; + label: string; +} + +export interface HistogramPreviewData { + bins: string[]; + counts: Array; + variableName: string; + variableType?: string; + groups?: HistogramGroupSeries[]; + splitLabel?: string; +} + +export interface PreprocessingRule { + variableCode: string; + action: MissingAction; + value: string; + enabled: boolean; +} + +export interface PreprocessingGroup { + key: 'pending' | 'applied' | 'default' | 'not-applied'; + title: string; + variables: VariableRow[]; +} + +export interface OutlierPreviewRow { + variable: string; + dataset: string; + strategy: string; + tail: string; + fold: string; + lowerBound: string; + upperBound: string; + lowerOutliers: string; + upperOutliers: string; + totalOutliers: string; + outlierPercentage: string; +} + +export interface DescriptiveProgressState { + pendingChangeCount: number; + preprocessingStatus: PreprocessingStatus; + transformationStatusLabel: string; +} +/** Copy of one "How to use" strip: numbered steps plus a closing note. */ +export interface HowToGuide { + steps: string[]; + note: string; +} diff --git a/src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.pipeline.spec.ts b/src/app/pages/experiment-studio/data-handling/data-handling.pipeline.spec.ts similarity index 94% rename from src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.pipeline.spec.ts rename to src/app/pages/experiment-studio/data-handling/data-handling.pipeline.spec.ts index d277390cb..8795cf0a2 100644 --- a/src/app/pages/experiment-studio/statistic-analysis-panel/statistic-analysis-panel.pipeline.spec.ts +++ b/src/app/pages/experiment-studio/data-handling/data-handling.pipeline.spec.ts @@ -3,7 +3,10 @@ import { ComponentFixture, TestBed } from '@angular/core/testing'; import { of, throwError } from 'rxjs'; import { provideZonelessChangeDetection, signal } from '@angular/core'; import { provideEchartsCore } from 'ngx-echarts'; -import { StatisticAnalysisPanelComponent } from './statistic-analysis-panel.component'; +import { DataHandlingComponent } from './data-handling.component'; +import { RuleBuilder } from './transformation/rule-builder-registry'; +import { emptySummaryView } from './utils/summary-view.utils'; +import type { SummaryView } from './data-handling.models'; import { ExperimentStudioService } from '../../../services/experiment-studio.service'; import { ExperimentsDashboardService } from '../../../services/experiments-dashboard.service'; import { ChartBuilderService } from '../visualisations/charts/chart-builder.service'; @@ -15,9 +18,29 @@ import { createSpyObj } from '../../../testing/spy-obj.testing'; * visit or present in the persisted request config. Opening a stage must never * write that config. */ -describe('StatisticAnalysisPanelComponent pipeline presence', () => { - let component: StatisticAnalysisPanelComponent; - let fixture: ComponentFixture; + +/** A stand-in rule builder: what it would export, and the error it would show. */ +interface FakeRuleBuilder { + exportFilterLogic: () => unknown; + filterError: () => string | null; +} + +/** + * Stands in for the rendered category builders: `builders[i]` edits the i-th rule in + * card order; rules past the end of the list have no builder (as if not rendered). + */ +function useRuleBuilders(component: DataHandlingComponent, builders: readonly FakeRuleBuilder[]): void { + component.transformationDrafts.flatMap((draft) => draft.rules).forEach((rule, index) => { + const live = component.ruleBuilders.get(rule); + if (live) component.ruleBuilders.unregister(rule, live); + const fake = builders[index]; + if (fake) component.ruleBuilders.register(rule, { ensureFirstCondition: () => undefined, ...fake } as RuleBuilder); + }); +} + +describe('DataHandlingComponent pipeline presence', () => { + let component: DataHandlingComponent; + let fixture: ComponentFixture; let mockExpService: MockedObject; const age = { code: 'age', label: 'Age', type: 'real' }; @@ -174,7 +197,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { seedAppliedConfig(config as Record | null)); await TestBed.configureTestingModule({ - imports: [StatisticAnalysisPanelComponent], + imports: [DataHandlingComponent], providers: [ provideZonelessChangeDetection(), provideEchartsCore({ echarts: () => import('echarts') }), @@ -185,7 +208,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { ] }).compileComponents(); - fixture = TestBed.createComponent(StatisticAnalysisPanelComponent); + fixture = TestBed.createComponent(DataHandlingComponent); component = fixture.componentInstance; fixture.detectChanges(); }); @@ -237,7 +260,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { expect(mockExpService.setAppliedDescriptivePreprocessing).not.toHaveBeenCalled(); expect(mockExpService.upsertTransformation).not.toHaveBeenCalled(); - expect((component as any).userPreprocessingApplied).toBe(false); + expect(component.preprocessingState.userApplied).toBe(false); }); it('persists on Apply and clears the persisted config on Remove', () => { @@ -559,9 +582,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { // Echo the hydrated filters back so committing does not blank them. const committed = component.transformationDrafts.flatMap((draft) => draft.rules.map((rule) => rule.filter)); - component.transformationRuleModals = { - toArray: () => committed.map((filter) => ({ exportFilterLogic: () => filter, filterError: () => null })), - } as any; + useRuleBuilders(component, committed.map((filter) => ({ exportFilterLogic: () => filter, filterError: () => null }))); // Every card answers for itself: its own Apply commits it and folds it, and the // stage stays open until a click on its header folds it to the overview. @@ -811,7 +832,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { it('closes a settled column card from its own footer without removing the step', () => { component.addStep('transformation'); // The stage opens on its type chooser; the footer bar belongs to the column card. - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); fixture.detectChanges(); const apply = (fixture.nativeElement as HTMLElement).querySelector( @@ -906,7 +927,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { it('shows category counts on the rule rows and has no preview button', () => { component.addStep('transformation'); - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); fixture.detectChanges(); const footer = (fixture.nativeElement as HTMLElement).querySelector( @@ -932,7 +953,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { it('numbers the column editor and adds categories from the ghost row', () => { component.goToSection('transformation'); // The stage opens on the type chooser; the editor belongs to the column card. - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); fixture.detectChanges(); const section = (fixture.nativeElement as HTMLElement).querySelector('[data-guide="analysis-transformation"]'); @@ -964,7 +985,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { // The rule builder renders its conditions only once there are variables to filter on. (mockExpService.selectedVariables as any).set([age]); component.goToSection('transformation'); - component.chooseTransformation('categorical'); + component.chooseTransformation('categorical_column_creator'); const draft = component.transformationDrafts[0]; component.addTransformationRule(draft); component.addTransformationRule(draft); @@ -995,7 +1016,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { .toBe('Editing category 2 of 2'); const filter = { condition: 'AND', rules: [{ id: 'age', field: 'age', operator: 'greater', value: 60 }] }; - const modal = component.transformationRuleModals!.toArray()[1]; + const modal = component.ruleBuilders.get(draft.rules[1])!; vi.spyOn(modal, 'exportFilterLogic').mockReturnValue(filter as any); (editing[0].querySelector('.transformation-rule-done') as HTMLButtonElement).click(); fixture.detectChanges(); @@ -1011,7 +1032,7 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { component.toggleTransformationRule(draft, 0); fixture.detectChanges(); - const modal = component.transformationRuleModals!.toArray()[0]; + const modal = component.ruleBuilders.get(draft.rules[0])!; vi.spyOn(modal, 'exportFilterLogic').mockReturnValue(null); vi.spyOn(modal, 'filterError').mockReturnValue('Pick a value'); component.finishTransformationRule(draft, 0); @@ -1023,23 +1044,23 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { describe('review regressions', () => { // Built on the component's own empty summary so teardown finds its histogram maps. - const summaryOf = (total: number) => ({ - ...(component as any).createEmptySummary(false), - featurewiseRows: [{ dataset: 'all datasets', data: { num_total: total } }], + const summaryOf = (total: number): SummaryView => ({ + ...emptySummaryView(false), + featurewiseRows: [{ dataset: 'all datasets', data: { num_total: total } }] as unknown as SummaryView['featurewiseRows'], }); it('keeps a chosen K-means card in the graph when it is folded', () => { component.addStep('transformation'); - component.chooseTransformation('kmeans'); + component.chooseTransformation('kmeans_cluster_creator'); component.toggleCardOpen(component.cards()[0].id, false); - expect(component.kmeansCardShown).toBe(true); + expect(component.cards().some((step) => step.name === 'kmeans_cluster_creator')).toBe(true); expect(component.transformationChooserOnly).toBe(false); }); it('Clear drops the K-means card and its preview, back to the chooser', () => { component.addStep('transformation'); - component.chooseTransformation('kmeans'); + component.chooseTransformation('kmeans_cluster_creator'); const stepId = component.cards().find((step) => step.name === 'kmeans_cluster_creator')!.id; mockExpService.upsertTransformation({ id: stepId, @@ -1060,16 +1081,16 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { component.resetTransformation(); expect(component.previewDraftId).toBeNull(); - expect(component.kmeansCardShown).toBe(false); + expect(component.cards().some((step) => step.name === 'kmeans_cluster_creator')).toBe(false); expect(component.transformationChooserOnly).toBe(true); }); it('previewing the K-means card commits no categorical card', () => { component.addStep('transformation'); - component.chooseTransformation('kmeans'); + component.chooseTransformation('kmeans_cluster_creator'); const stepId = component.cards().find((step) => step.name === 'kmeans_cluster_creator')!.id; const commit = vi.spyOn(component, 'commitTransformationRuleFilters').mockReturnValue(undefined); - vi.spyOn(component, 'refreshTransformationStatistics').mockReturnValue(undefined); + vi.spyOn(component.transformationStats, 'refresh').mockReturnValue(undefined); component.previewTransformationCard(stepId); @@ -1079,26 +1100,26 @@ describe('StatisticAnalysisPanelComponent pipeline presence', () => { it('hides Records out while the Raw summary is pinned to an unapplied preview', () => { (mockExpService.filterLogic as any).set(null); - (component as any).rawSummaryKey = (component as any).rawSummaryKeyFor(); - (component as any).rawSummary = summaryOf(400); + component.statisticSummary.loadedKey.raw = component.statisticSummary.rawKeyFor(); + component.rawSummary = summaryOf(400); expect(component.recordsOut.source).toBe(400); expect(component.recordsOut.filtered).toBe(400); // Pinned to unsaved "age >= 60": 120 rows are not the source cohort. const pinned = { condition: 'AND', rules: [{ id: 'age', field: 'age', operator: 'greater_or_equal', value: 60 }] } as any; - (component as any).rawSummaryKey = (component as any).rawSummaryKeyFor(pinned); - (component as any).rawSummary = summaryOf(120); + component.statisticSummary.loadedKey.raw = component.statisticSummary.rawKeyFor(pinned); + component.rawSummary = summaryOf(120); expect(component.recordsOut.source).toBeNull(); expect(component.recordsOut.filtered).toBeNull(); }); it('never rounds a partial filter match to 0% or 100%', () => { - (component as any).sourceSummary = summaryOf(1000); - (component as any).rawSummary = summaryOf(998); + component.sourceSummary = summaryOf(1000); + component.rawSummary = summaryOf(998); expect(component.filterPreviewStatus).toContain('· 99%'); - (component as any).rawSummary = summaryOf(2); + component.rawSummary = summaryOf(2); expect(component.filterPreviewStatus).toContain('· 1%'); - (component as any).rawSummary = summaryOf(1000); + component.rawSummary = summaryOf(1000); expect(component.filterPreviewStatus).toContain('· 100%'); }); diff --git a/src/app/pages/experiment-studio/data-handling/data-handling.preprocessing.css b/src/app/pages/experiment-studio/data-handling/data-handling.preprocessing.css new file mode 100644 index 000000000..55038249b --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/data-handling.preprocessing.css @@ -0,0 +1,37 @@ +/* Preprocessing stations of the Data review step: station shell, preview bar, + glass feedback cards, the documentation panel, the missing/outlier/longitudinal + controls and the preprocessing variable browser. + Split from data-handling.component.css to keep every component + stylesheet inside the anyComponentStyle budget. Angular concatenates the + component's styleUrl array IN ORDER, and this file is the second of three — + the order below is the cascade, so do not reorder or move rules across files: + 1. data-handling.component.css panel shell + node-pipeline canvas + 2. data-handling.preprocessing.css (this file) + 3. data-handling.results.css shared controls + result surfaces + Shared/one-off overrides that must win over styles.css live in the file that + owns the rule's section, never duplicated. */ + +@import './styles/search-and-empty-states.shared.css'; + +/* Stations sit in a gapped shell: the preview bar needs no extra bottom margin. */ +.preprocessing-shell .station-preview-bar { + margin-bottom: 0; +} + +.empty-state { + margin: 12px 0; + text-align: center; + font-size: 0.9rem; + color: var(--text-muted); + padding: 24px; + background: var(--studio-card-header-bg); + border: 1px solid var(--studio-card-border, rgba(43, 51, 233, 0.1)); + box-shadow: none; + border-radius: var(--radius-lg, 12px); + display: flex; + flex-direction: column; + align-items: center; + gap: 12px; + min-height: 160px; + justify-content: center; +} diff --git a/src/app/pages/experiment-studio/data-handling/data-handling.results.css b/src/app/pages/experiment-studio/data-handling/data-handling.results.css new file mode 100644 index 000000000..7e18cfc15 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/data-handling.results.css @@ -0,0 +1,52 @@ +/* Shared controls and result surfaces of the Data review step: the toggle + switch, the federated-data note, the descriptive/data tables, the statistics + browser, the chart-export-only containers, the transformation station and the + responsive overrides for all three files. + Second split of data-handling.component.css (see + data-handling.preprocessing.css for the reason and the cascade + order). This file is LAST in the styleUrl array, which is why the responsive + overrides live here. */ + +@import './styles/tables.shared.css'; + +/* Rules below are prefixed with .statistics-table-wrapper so they outrank the + --numeric overrides above: emulated encapsulation adds one attribute per + compound selector, so a shorter selector loses regardless of order. */ + +.hidden-charts-for-export { + position: fixed; + top: -9999px; + left: -9999px; + width: 900px; + height: auto; + opacity: 1; + pointer-events: none; + background: var(--card-bg); +} + +.hidden-charts-for-export app-chart-renderer { + width: 800px; + height: 500px; + display: block; +} + +.pdf-overlay { + position: absolute; + inset: 0; + background: rgba(255, 255, 255, 0.8); + display: flex; + flex-direction: column; + justify-content: center; + align-items: center; + z-index: 999; + font-size: 1.2rem; + color: #333; + font-weight: 500; +} + +@media (max-width: 768px) { + .modal-content { + border-radius: var(--radius-lg, 12px); + } + +} diff --git a/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.css b/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.css new file mode 100644 index 000000000..dbb800753 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.css @@ -0,0 +1,110 @@ +/* The panel nudges every icon 1px down (`.stats-panel-shell i`); the rule is + scoped to the panel's own template, so it is repeated here for this one. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} + +/* "How to use" strip directly under an expanded stage header; a
    folded by + default, so only the summary line shows until someone asks for the steps. The +
    is this component's host, so the strip adds no wrapper element. */ +:host(.how-to-strip) { + background: color-mix(in srgb, var(--primary-color) 4%, white); + border-bottom: 1px solid rgba(43, 51, 233, 0.08); +} + +.how-to-summary { + display: flex; + align-items: center; + gap: 10px; + padding: 8px 18px; + list-style: none; + cursor: pointer; +} + +.how-to-summary::-webkit-details-marker { + display: none; +} + +.how-to-summary:hover .how-to-peek { + color: var(--primary-color); +} + +.how-to-summary:focus-visible { + outline: 2px solid var(--primary-color); + outline-offset: -2px; +} + +.how-to-label { + font-size: 0.66rem; + font-weight: 700; + letter-spacing: 0.06em; + text-transform: uppercase; + color: var(--primary-color); +} + +.how-to-peek { + font-size: 0.74rem; + color: var(--text-muted); +} + +.how-to-chevron { + font-size: 0.6rem; + color: var(--primary-color); + transition: transform 0.15s ease; +} + +:host(.how-to-strip[open]) .how-to-chevron { + transform: rotate(180deg); +} + +:host(.how-to-strip[open]) .how-to-peek { + display: none; +} + +.how-to-body { + display: flex; + flex-wrap: wrap; + align-items: center; + gap: 8px 14px; + padding: 0 18px 10px; +} + +.how-to-step { + display: inline-flex; + align-items: center; + gap: 7px; + font-size: 0.78rem; + color: #334155; +} + +.how-to-number { + display: inline-flex; + align-items: center; + justify-content: center; + flex: none; + width: 18px; + height: 18px; + border-radius: 50%; + background: var(--primary-color); + color: #fff; + font-size: 0.68rem; + font-weight: 700; +} + +.how-to-note { + flex-basis: 100%; + font-size: 0.76rem; + color: var(--text-muted); +} + +/* Projected into a station card, the strip spans the card body edge to edge. + The card belongs to the Data Handling template, so it is matched as host context. */ +:host-context(app-station-card) { + margin: -14px -14px 14px; +} + +@media (max-width: 768px) { + :host-context(app-station-card) { + margin: -14px -12px 14px; + } +} diff --git a/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.html b/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.html new file mode 100644 index 000000000..e23974f30 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.html @@ -0,0 +1,11 @@ + + How to use + {{ guide().steps.length }} steps + + +
    + @for (step of guide().steps; track step; let i = $index) { + {{ i + 1 }}{{ step }} + } + {{ guide().note }} +
    diff --git a/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.ts b/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.ts new file mode 100644 index 000000000..8d64d6848 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/how-to-strip/how-to-strip.component.ts @@ -0,0 +1,19 @@ +import { ChangeDetectionStrategy, Component, input } from '@angular/core'; +import { HowToGuide } from '../data-handling.models'; + +/** + * "How to use" strip under an expanded stage header: numbered steps plus one note. + * Folded by default so it never pushes the editor down; the summary line opens it. + * Attached to the
    element itself, so the DOM is the same as when it was an + * inline template and parent rules such as `.workflow-section-body.open > *` still reach it. + */ +@Component({ + selector: 'details[app-how-to-strip]', + host: { class: 'how-to-strip' }, + templateUrl: './how-to-strip.component.html', + styleUrl: './how-to-strip.component.css', + changeDetection: ChangeDetectionStrategy.OnPush, +}) +export class HowToStripComponent { + readonly guide = input.required(); +} diff --git a/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.css b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.css new file mode 100644 index 000000000..a416d6bc1 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.css @@ -0,0 +1,157 @@ +/* The panel nudges every icon 1px down (`.stats-panel-shell i`); the rule is + scoped to the panel's own template, so it is repeated here for this one. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} + +/* Centred and capped like a collapsed pipeline step (`.pipeline-node.is-compact`). */ +:host(.pipeline-terminal-card) { + align-self: center; + width: 100%; + max-width: var(--studio-pipeline-overview-max, 860px); +} + +/* Terminal Card: this component's host. */ +:host(.pipeline-terminal-card) { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + align-items: center; + padding: 16px 20px; + background: var(--studio-card-header-bg); + border: 1px solid var(--border-color, rgba(43, 51, 233, 0.15)); + border-radius: var(--radius-lg, 12px); + box-shadow: none; + gap: 20px; + margin-top: 4px; +} + +.pipeline-terminal-summary { + display: flex; + flex-direction: column; + gap: 8px; + min-width: 0; +} + +.pipeline-terminal-kicker { + color: var(--text-muted); + font-size: 0.72rem; + font-weight: 700; + letter-spacing: var(--letter-spacing-config); + text-transform: uppercase; +} + +.pipeline-terminal-facts { + display: flex; + flex-wrap: wrap; + align-items: baseline; + gap: 6px 22px; + font-size: 0.84rem; + color: var(--text-muted); +} + +.pipeline-terminal-facts strong { + font-size: 1.05rem; + font-weight: 700; + color: var(--text-main); +} + +.btn-finish-pipeline { + display: inline-flex; + align-items: center; + gap: 8px; + padding: 10px 22px; + background: var(--primary-color); + color: #ffffff; + border: none; + border-radius: 999px; + font-family: inherit; + font-size: 0.86rem; + font-weight: 600; + cursor: pointer; + box-shadow: var(--shadow-sm); + transition: all 0.2s ease; + white-space: nowrap; +} + +/* Unapplied-changes guard */ +.unapplied-warning-card { + grid-column: 1 / -1; + flex: 1 1 100%; + display: flex; + align-items: center; + gap: 14px; + padding: 12px 16px; + background: color-mix(in srgb, #f59e0b 8%, white); + border: 1px solid color-mix(in srgb, #f59e0b 30%, transparent); + border-radius: var(--radius-md, 8px); +} + +.unapplied-warning-icon { + color: #78350f; + font-size: 1.1rem; + flex-shrink: 0; +} + +.unapplied-warning-text h4 { + font-size: 0.88rem; + font-weight: 700; + color: var(--text-main); + margin: 0 0 2px; +} + +.unapplied-warning-text p { + font-size: 0.78rem; + color: var(--text-muted); + margin: 0; +} + +.unapplied-warning-actions { + margin-left: auto; + display: flex; + gap: 8px; + align-items: center; +} + +/* Secondary and ghost siblings of the primary pill: same shape, no fill. The frame is + an inset shadow so the box stays exactly as tall as .btn-finish-pipeline (border: none). */ +.unapplied-warning-btn.secondary { + background: var(--card-bg); + color: var(--primary-color); + box-shadow: inset 0 0 0 1px color-mix(in srgb, var(--primary-color) 35%, transparent); +} + +.unapplied-warning-btn.ghost { + background: transparent; + color: var(--text-muted); + box-shadow: none; +} + +.unapplied-warning-btn.secondary:hover, +.unapplied-warning-btn.ghost:hover { + transform: none; +} + +.unapplied-warning-btn.secondary:hover { + background: color-mix(in srgb, var(--primary-color) 8%, white); +} + +.unapplied-warning-btn.ghost:hover { + background: transparent; + color: var(--text-main); +} + +.btn-finish-pipeline:hover { + background: var(--primary-dark, #1b21a3); + transform: none; + box-shadow: var(--shadow-md); +} + +@media (max-width: 768px) { + :host(.pipeline-terminal-card) { + grid-template-columns: minmax(0, 1fr); + align-items: stretch; + } + .btn-finish-pipeline { + justify-content: center; + } +} diff --git a/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.html b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.html new file mode 100644 index 000000000..4763de382 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.html @@ -0,0 +1,35 @@ +@if (showUnappliedWarning()) { + +} +
    + The next run will use +
    + @if (retention(); as r) { + {{ r.processed.toLocaleString() }} of {{ r.raw.toLocaleString() }} records + } + {{ variableCount() }} variables · {{ datasetCount() }} datasets + {{ preprocessingLine() }} +
    +
    + diff --git a/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.spec.ts b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.spec.ts new file mode 100644 index 000000000..214dbeb31 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.spec.ts @@ -0,0 +1,58 @@ +import { Component, signal } from '@angular/core'; +import { TestBed } from '@angular/core/testing'; +import { PipelineTerminalComponent } from './pipeline-terminal.component'; + +@Component({ + imports: [PipelineTerminalComponent], + template: `
    `, +}) +class HostComponent { + readonly retention = signal<{ raw: number; processed: number } | null>({ raw: 1200, processed: 1000 }); + readonly warn = signal(false); + readonly log: string[] = []; +} + +describe('PipelineTerminalComponent', () => { + async function render() { + const fixture = TestBed.createComponent(HostComponent); + await fixture.whenStable(); + return { fixture, host: fixture.componentInstance, el: fixture.nativeElement as HTMLElement }; + } + + it('renders on the card element itself with the run summary', async () => { + const { el } = await render(); + const card = el.querySelector('div.pipeline-terminal-card')!; + expect(card).toBeTruthy(); + expect(card.textContent).toContain('1,000 of 1,200 records'); + expect(card.textContent).toContain('3 variables · 2 datasets'); + expect(card.textContent).toContain('No preprocessing'); + expect(card.querySelector('.unapplied-warning-card')).toBeNull(); + }); + + it('hides the records line until retention is known', async () => { + const { fixture, host, el } = await render(); + host.retention.set(null); + await fixture.whenStable(); + expect(el.textContent).not.toContain('records'); + }); + + it('emits continue from the hand-off button', async () => { + const { host, el } = await render(); + (el.querySelector('.pipeline-terminal-card > button.btn-finish-pipeline') as HTMLButtonElement).click(); + expect(host.log).toEqual(['continue']); + }); + + it('shows the unapplied-changes guard and emits each choice', async () => { + const { fixture, host, el } = await render(); + host.warn.set(true); + await fixture.whenStable(); + const buttons = el.querySelectorAll('.unapplied-warning-actions button'); + expect(el.querySelector('[role="alert"]')?.textContent).toContain('You have unapplied changes'); + buttons.forEach((b) => b.click()); + expect(host.log).toEqual(['apply', 'discard', 'stay']); + }); +}); diff --git a/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.ts b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.ts new file mode 100644 index 000000000..3e210333e --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/pipeline-terminal/pipeline-terminal.component.ts @@ -0,0 +1,27 @@ +import { ChangeDetectionStrategy, Component, input, output } from '@angular/core'; + +/** + * End of the Data Handling pipeline: what the next run will use, the hand-off to + * Algorithm Selection, and the guard shown when edits are still unapplied. + * Attached to the card element itself, so the pipeline column lays it out as before. + */ +@Component({ + selector: 'div[app-pipeline-terminal]', + host: { class: 'pipeline-terminal-card' }, + templateUrl: './pipeline-terminal.component.html', + styleUrl: './pipeline-terminal.component.css', + changeDetection: ChangeDetectionStrategy.OnPush, +}) +export class PipelineTerminalComponent { + /** Records kept after preprocessing out of the raw cohort; null until both are known. */ + readonly retention = input<{ raw: number; processed: number } | null>(null); + readonly variableCount = input(0); + readonly datasetCount = input(0); + readonly preprocessingLine = input(''); + readonly showUnappliedWarning = input(false); + + readonly continue = output(); + readonly applyAndContinue = output(); + readonly discardAndContinue = output(); + readonly stay = output(); +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.css b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.css new file mode 100644 index 000000000..62f055157 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.css @@ -0,0 +1,11 @@ +/* The longitudinal step has no styles of its own beyond the shared step styles. */ + +@import '../../styles/statistics-panel.shared.css'; +@import '../../styles/search-and-empty-states.shared.css'; +@import '../../styles/tables.shared.css'; +@import '../../styles/preprocessing-step.shared.css'; + +/* The panel's 1px icon nudge (`.stats-panel-shell i`), repeated for this template. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.html b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.html new file mode 100644 index 000000000..809f740bd --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.html @@ -0,0 +1,55 @@ + + +
    + @if (dh.selectedPrepVariable('longitudinal'); as variable) { + @let state = dh.prepVariableStateLabel('longitudinal', variable); +
    +
    + {{ dh.variableTypeLabel(variable) }} variable +

    {{ dh.variableLabel(variable) }}

    +
    + + {{ state }} + +
    + +
    +
    + + Longitudinal strategy +
    + + @if (dh.preprocessingValidationErrors[variable.code]) { +
    {{ dh.preprocessingValidationErrors[variable.code] }}
    + } +
    + } @else { +
    + +

    No variables match the current search.

    + Clear the search field to choose a longitudinal strategy variable. +
    + } +
    diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.spec.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.spec.ts new file mode 100644 index 000000000..98a558ec0 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.spec.ts @@ -0,0 +1,61 @@ +import { Component } from '@angular/core'; +import { TestBed } from '@angular/core/testing'; +import { vi } from 'vitest'; +import type { DataHandlingComponent } from '../../data-handling.component'; +import { DATA_HANDLING } from '../../data-handling.context'; +import type { LongitudinalStrategy, VariableRow } from '../../data-handling.models'; +import { LongitudinalStepComponent } from './longitudinal-step.component'; + +@Component({ + imports: [LongitudinalStepComponent], + template: `
    `, +}) +class HostComponent {} + +const age = { code: 'age', label: 'Age' } as VariableRow; + +function render(error: string | null = null) { + const dh = { + prepSearch: { missing: '', outlier: '', longitudinal: '' }, + prepGroups: () => [{ key: 'applied', title: 'Applied preprocessing', variables: [age] }], + filteredPrepVariables: () => [age], + isPrepVariableSelected: () => true, + selectPrepVariable: vi.fn(), + selectedPrepVariable: () => age, + variableLabel: (variable: VariableRow) => variable.label, + variableTypeLabel: () => 'Numerical', + prepVariableStateLabel: () => 'Applied', + prepVariableHasPendingChange: () => false, + longitudinalStrategies: [ + { value: 'first', label: 'Use visit 1' }, + { value: 'diff', label: 'Difference' }, + ], + longitudinalStrategyFor: () => 'first', + isLongitudinalStrategyDisabled: (_variable: VariableRow, strategy: LongitudinalStrategy) => strategy === 'diff', + onLongitudinalStrategyChange: vi.fn(), + preprocessingValidationErrors: (error ? { age: error } : {}) as Record, + }; + TestBed.configureTestingModule({ providers: [{ provide: DATA_HANDLING, useValue: dh as unknown as DataHandlingComponent }] }); + const fixture = TestBed.createComponent(HostComponent); + fixture.detectChanges(); + return { dh, el: fixture.nativeElement as HTMLElement }; +} + +describe('LongitudinalStepComponent', () => { + it('renders a rail without group states and the type label in each row', () => { + const { el } = render(); + expect(el.querySelector('.preprocessing-variable-group')?.hasAttribute('data-state')).toBe(false); + expect(el.querySelector('.statistics-variable-meta small')?.textContent).toBe('Numerical'); + expect(el.querySelector('.preprocessing-state-chip')?.className).toContain('state-applied'); + }); + + it('offers the strategies with the unavailable ones disabled', () => { + const { el } = render(); + expect(el.querySelector('.preprocessing-control > span')?.textContent).toBe('Strategy for Age'); + const options = Array.from(el.querySelectorAll('select.table-select option')); + expect(options.map((option) => [option.textContent, option.disabled])).toEqual([ + ['Use visit 1', false], + ['Difference', true], + ]); + }); +}); diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.ts new file mode 100644 index 000000000..547845568 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/longitudinal-step/longitudinal-step.component.ts @@ -0,0 +1,21 @@ +import { ChangeDetectionStrategy, Component, inject } from '@angular/core'; +import { FormsModule } from '@angular/forms'; +import { DATA_HANDLING } from '../../data-handling.context'; +import { PrepRailSlotDirective, PrepVariableRailComponent } from '../prep-variable-rail/prep-variable-rail.component'; + +/** + * The longitudinal sub-step workspace: the variable rail and the detail panel with the + * per-variable comparison strategy. + * Attached to the step's workspace
    , so the DOM and selectors are unchanged. + */ +@Component({ + selector: 'section[app-longitudinal-step]', + imports: [FormsModule, PrepVariableRailComponent, PrepRailSlotDirective], + templateUrl: './longitudinal-step.component.html', + styleUrl: './longitudinal-step.component.css', + // Reads the pipeline's plain fields, so it refreshes with the pipeline (see DATA_HANDLING). + changeDetection: ChangeDetectionStrategy.Eager, +}) +export class LongitudinalStepComponent { + protected readonly dh = inject(DATA_HANDLING); +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.css b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.css new file mode 100644 index 000000000..1a8776a95 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.css @@ -0,0 +1,227 @@ +/* The missing-values step's own styles (action choices, coverage tiles, the frequency + list), in the step's cascade order. */ + +@import '../../styles/statistics-panel.shared.css'; +@import '../../styles/search-and-empty-states.shared.css'; +@import '../../styles/tables.shared.css'; +@import '../../styles/preprocessing-step.shared.css'; + +/* The panel's 1px icon nudge (`.stats-panel-shell i`), repeated for this template. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} + +/* The winsorizer carries four controls, so it needs the wider editor column. */ + +.preprocessing-header-coverage { + color: var(--text-muted); + font-size: 0.76rem; + font-variant-numeric: tabular-nums; + white-space: nowrap; +} + +.preprocessing-evidence-group { + display: flex; + flex-direction: column; + align-items: flex-start; + gap: 8px; + min-width: 0; +} + +.preprocessing-stat-grid { + display: grid; + gap: 8px 14px; + margin: 0; +} + +.preprocessing-stat-grid { + grid-template-columns: repeat(3, minmax(0, 1fr)); +} + +/* Filled coverage meter — an outline-only gauge would not encode the value. + --success is only 2.07:1 against this track, so a 0%-missing row renders an + empty track (which is the correct reading) rather than a pale green nub. */ +.preprocessing-meter { + display: block; + height: 6px; + border-radius: 999px; + background: color-mix(in srgb, var(--text-muted) 14%, white); + overflow: hidden; +} + +.preprocessing-meter-fill { + display: block; + height: 100%; + border-radius: 999px; +} + +/* Keep a non-zero share visible even when it is tiny. */ +.preprocessing-evidence[data-severity='warning'] .preprocessing-meter-fill, +.preprocessing-evidence[data-severity='high'] .preprocessing-meter-fill { + min-width: 6px; +} + +.preprocessing-evidence[data-severity='warning'] .preprocessing-meter-fill { + /* --accent-color alone is 1.9:1 on the track; deepened with the brand's own + filter brown so the fill clears 3:1 without inventing a new hex. */ + background: color-mix(in srgb, var(--accent-color) 42%, var(--filter-color)); +} + +.preprocessing-evidence[data-severity='high'] .preprocessing-meter-fill { + background: var(--danger); +} + +.preprocessing-evidence-group { + display: grid; + grid-template-columns: minmax(0, 1fr); + align-content: start; + gap: 8px; + min-width: 0; +} + +.preprocessing-evidence-wide { + grid-column: 1 / -1; +} + +.preprocessing-frequency-list { + display: grid; + gap: 6px; + margin: 0; + padding: 0; + list-style: none; +} + +.preprocessing-frequency { + display: grid; + grid-template-columns: minmax(0, 1fr) minmax(48px, 2fr) auto; + align-items: center; + gap: 10px; +} + +.preprocessing-frequency-label { + color: var(--text-main); + font-size: 0.8125rem; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.preprocessing-frequency-track { + height: 6px; + border-radius: 999px; + background: color-mix(in srgb, var(--text-muted) 14%, white); + overflow: hidden; +} + +.preprocessing-frequency-bar { + display: block; + height: 100%; + min-width: 4px; + border-radius: 999px; + background: var(--primary-color); +} + +.preprocessing-frequency-value { + color: var(--text-main); + font-size: 0.8125rem; + font-weight: 600; + font-variant-numeric: tabular-nums; +} + +@media (max-width: 900px) { + .preprocessing-stat-grid { + grid-template-columns: repeat(3, minmax(0, 1fr)); + } +} + +/* 2×2 so each card's effect sentence stays readable. */ +.missing-choice-grid { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 8px; + margin: 0; + padding: 0; + border: 0; +} + +/* Missing-values choice cards (replaces the action select). */ +.missing-choice-grid legend { + grid-column: 1 / -1; + margin-bottom: 6px; + padding: 0; +} + +.missing-choice { + position: relative; + display: grid; + gap: 4px; + padding: 10px 12px 10px 34px; + border-radius: var(--radius-md); + background: var(--card-bg); + box-shadow: inset 0 0 0 1px var(--studio-card-border); + cursor: pointer; +} + +.missing-choice:hover { + background: #f7f8fe; +} + +.missing-choice.is-selected { + background: color-mix(in srgb, var(--primary-color) 4%, white); + box-shadow: inset 0 0 0 1.5px var(--primary-color); +} + +.missing-choice.is-disabled { + opacity: 0.5; + cursor: not-allowed; +} + +.missing-choice input[type='radio'] { + position: absolute; + left: 12px; + top: 13px; +} + +.missing-choice-label { + font-size: 0.84rem; + font-weight: 600; + color: var(--text-main); +} + +.missing-choice-effect { + font-size: 0.74rem; + line-height: 1.4; + color: var(--text-muted); + text-wrap: pretty; +} + +/* Per-dataset coverage tiles. */ +.coverage-tiles { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(160px, 1fr)); + gap: 10px; + width: 100%; +} + +.coverage-tile { + display: grid; + gap: 6px; + padding: 10px 12px; + border-radius: var(--radius-md); + background: var(--bg-neutral); +} + +.coverage-tile-head { + display: flex; + justify-content: space-between; + gap: 8px; + font-size: 0.76rem; +} + +.coverage-tile-head strong { + font-weight: 700; +} + +.coverage-tile-head span { + color: var(--text-muted); +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.html b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.html new file mode 100644 index 000000000..d86bf4db9 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.html @@ -0,0 +1,202 @@ + + +
    + @if (dh.selectedPrepVariable('missing'); as variable) { + @let rule = dh.ruleFor(variable); + @let state = dh.prepVariableStateLabel('missing', variable); +
    +
    + {{ dh.variableTypeLabel(variable) }} variable +

    {{ dh.variableLabel(variable) }} @if (variable.code) {{{ variable.code }}}

    +
    + @if (dh.coverageFor(variable); as headerCoverage) { + {{ headerCoverage.missing.toLocaleString() }} of {{ headerCoverage.total.toLocaleString() }} values missing + } + + {{ state }} + +
    + +
    +
    + + Missing value handling +
    +
    + What to do with missing values + @for (action of dh.missingActions; track action.value) { + + } +
    + @if (rule.action === 'constant') { + + } + @if (dh.preprocessingValidationErrors[variable.code]) { +
    {{ dh.preprocessingValidationErrors[variable.code] }}
    + } +
    + + @if (dh.coverageFor(variable); as coverage) { + + } @else { + + + } + + } @else { +
    + +

    No variables match the current search.

    + Clear the search field to choose a preprocessing variable. +
    + } +
    diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.spec.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.spec.ts new file mode 100644 index 000000000..02ff7f0b2 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.spec.ts @@ -0,0 +1,93 @@ +import { Component } from '@angular/core'; +import { TestBed } from '@angular/core/testing'; +import { vi } from 'vitest'; +import type { DataHandlingComponent } from '../../data-handling.component'; +import { DATA_HANDLING } from '../../data-handling.context'; +import type { MissingAction, PreprocessingRule, VariableRow } from '../../data-handling.models'; +import { MissingValuesStepComponent } from './missing-values-step.component'; + +@Component({ + imports: [MissingValuesStepComponent], + template: `
    `, +}) +class HostComponent {} + +const sex = { code: 'sex', label: 'Sex' } as VariableRow; + +function render(rule: Partial, { categorical = true, coverage = null as { total: number; missing: number; share: number } | null } = {}) { + const current: PreprocessingRule = { variableCode: 'sex', action: 'drop', value: '', enabled: true, ...rule }; + const dh = { + // the rail + prepSearch: { missing: '', outlier: '', longitudinal: '' }, + prepGroups: () => [{ key: 'pending', title: 'Pending', variables: [sex] }], + filteredPrepVariables: () => [sex], + isPrepVariableSelected: () => true, + selectPrepVariable: vi.fn(), + batchMenuOpen: () => null, + toggleBatchMenu: vi.fn(), + applyBatchMissingStrategy: vi.fn(), + missingStrategyLabel: (action: MissingAction) => action, + // the detail panel + selectedPrepVariable: () => sex, + variableLabel: (variable: VariableRow) => variable.label, + variableTypeLabel: () => 'Categorical', + prepVariableStateLabel: () => 'Pending', + prepVariableHasPendingChange: () => true, + ruleFor: vi.fn(() => current), + missingActions: [ + { value: 'drop', label: 'Remove rows' }, + { value: 'constant', label: 'Fill with a value' }, + ], + isMissingActionDisabled: (_variable: VariableRow, action: MissingAction) => action === 'mean', + missingActionEffect: () => 'effect', + onMissingActionChange: vi.fn(), + onMissingValueChange: vi.fn(), + isCategoricalVariable: () => categorical, + enumOptions: () => [{ code: 'F', label: 'Female' }], + preprocessingValidationErrors: {} as Record, + coverageFor: () => coverage, + coverageSeverity: () => 'low', + coverageTitle: () => '', + coveragePercent: () => '10%', + coverageByDataset: () => [], + barWidth: () => 0, + distributionFor: () => [], + categoryCountsFor: () => [], + summaryUnavailableNote: () => 'Statistics are not available.', + }; + TestBed.configureTestingModule({ providers: [{ provide: DATA_HANDLING, useValue: dh as unknown as DataHandlingComponent }] }); + const fixture = TestBed.createComponent(HostComponent); + fixture.detectChanges(); + return { dh, el: fixture.nativeElement as HTMLElement }; +} + +describe('MissingValuesStepComponent', () => { + it('renders the rail and the selected variable with its action choices', () => { + const { dh, el } = render({ action: 'drop' }); + expect(el.querySelector('aside.preprocessing-browser .prep-rail-action')?.textContent?.trim()).toBe('drop · not applied'); + expect(el.querySelector('.preprocessing-state-chip')?.className).toContain('state-pending'); + const choices = Array.from(el.querySelectorAll('.missing-choice input')); + expect(choices.map((input) => input.checked)).toEqual([true, false]); + choices[1].dispatchEvent(new Event('change')); + expect(dh.onMissingActionChange).toHaveBeenCalledWith(sex, 'constant'); + expect(el.querySelector('.preprocessing-control')).toBeNull(); + }); + + it('asks for a category to fill with for a categorical variable, a free value otherwise', () => { + const categorical = render({ action: 'constant' }).el; + expect(categorical.querySelector('.preprocessing-control select.table-select')).not.toBeNull(); + TestBed.resetTestingModule(); + const numeric = render({ action: 'constant' }, { categorical: false }).el; + expect(numeric.querySelector('.preprocessing-control input.table-input')).not.toBeNull(); + }); + + it('says when statistics are unavailable, and shows the coverage when they are', () => { + expect(render({}).el.querySelector('.preprocessing-evidence[data-severity="none"] .preprocessing-consequence')?.textContent).toBe( + 'Statistics are not available.', + ); + TestBed.resetTestingModule(); + const el = render({}, { coverage: { total: 100, missing: 10, share: 0.1 } }).el; + expect(el.querySelector('.preprocessing-header-coverage')?.textContent).toBe('10 of 100 values missing'); + expect(el.querySelector('.preprocessing-evidence')?.getAttribute('data-severity')).toBe('low'); + }); +}); diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.ts new file mode 100644 index 000000000..ac9564c5b --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/missing-values-step/missing-values-step.component.ts @@ -0,0 +1,22 @@ +import { ChangeDetectionStrategy, Component, inject } from '@angular/core'; +import { FormsModule } from '@angular/forms'; +import { DATA_HANDLING } from '../../data-handling.context'; +import { PrepRailSlotDirective, PrepVariableRailComponent } from '../prep-variable-rail/prep-variable-rail.component'; + +/** + * The missing-values sub-step workspace: the variable rail (with coverage and the + * current action per row, and the "Set all" batch menu) and the detail panel with the + * action choice, fill value and the dataset coverage / distribution evidence. + * Attached to the step's workspace
    , so the DOM and selectors are unchanged. + */ +@Component({ + selector: 'section[app-missing-values-step]', + imports: [FormsModule, PrepVariableRailComponent, PrepRailSlotDirective], + templateUrl: './missing-values-step.component.html', + styleUrl: './missing-values-step.component.css', + // Reads the pipeline's plain fields, so it refreshes with the pipeline (see DATA_HANDLING). + changeDetection: ChangeDetectionStrategy.Eager, +}) +export class MissingValuesStepComponent { + protected readonly dh = inject(DATA_HANDLING); +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.css b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.css new file mode 100644 index 000000000..6cb473681 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.css @@ -0,0 +1,259 @@ +/* The outlier step's own styles (bounds controls and strip, the report table), in the + step's cascade order. */ + +@import '../../styles/statistics-panel.shared.css'; +@import '../../styles/search-and-empty-states.shared.css'; +@import '../../styles/tables.shared.css'; +@import '../../styles/preprocessing-step.shared.css'; + +/* The panel's 1px icon nudge (`.stats-panel-shell i`), repeated for this template. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} + +.outlier-preview-skeleton { + margin: 0; +} + +/* Mini variant of the studio switch, used in the outlier rail rows. */ +.toggle-switch.is-mini { + width: 30px; + height: 18px; +} + +.toggle-switch.is-mini .toggle-slider:before { + width: 14px; + height: 14px; + left: 1px; + top: 1px; +} + +.toggle-switch.is-mini input:checked + .toggle-slider:before { + transform: translateX(12px); +} + +/* Outlier bounds strip. Pooled over all datasets; the report below is per dataset. */ +.bounds-strip { + display: grid; + gap: 8px; + padding: 14px 16px 10px; + border-radius: var(--radius-md); + background: var(--bg-neutral); +} + +.bounds-labels, +.bounds-scale { + position: relative; + height: 18px; + font-size: 0.72rem; + color: var(--text-muted); +} + +.bounds-labels span { + position: absolute; + top: 0; + font-weight: 700; + color: var(--text-main); + white-space: nowrap; +} + +.bounds-track { + position: relative; + height: 44px; +} + +.bounds-track::before { + content: ''; + position: absolute; + inset: 20px 0 auto; + height: 4px; + border-radius: 999px; + background: #e5e7eb; +} + +.bounds-iqr { + position: absolute; + top: 12px; + height: 20px; + border-radius: var(--radius-sm); + background: rgba(43, 51, 233, 0.14); + box-shadow: inset 0 0 0 1px rgba(43, 51, 233, 0.35); +} + +.bounds-median { + position: absolute; + top: 12px; + width: 2px; + height: 20px; + background: var(--primary-color); +} + +.bounds-capped { + position: absolute; + top: 20px; + height: 4px; + background: color-mix(in srgb, var(--danger) 75%, var(--text-main)); +} + +.bounds-marker { + position: absolute; + top: 4px; + width: 2px; + height: 36px; + background: var(--text-main); +} + +.bounds-scale span { + position: absolute; + top: 0; + transform: translateX(-50%); + white-space: nowrap; +} + +.bounds-scale .is-min { + left: 0; + transform: none; +} + +.bounds-scale .is-max { + right: 0; + left: auto; + transform: none; +} + +.outlier-report-section { + display: grid; + gap: 10px; + margin-top: 4px; +} + +.outlier-report-header { + display: flex; + align-items: center; + justify-content: space-between; + gap: 12px; +} + +.outlier-report-table .share-cell { + display: flex; + align-items: center; + justify-content: flex-end; + gap: 8px; +} + +.share-bar { + display: inline-block; + width: 74px; + height: 4px; + border-radius: 999px; + background: rgba(43, 51, 233, 0.08); + overflow: hidden; +} + +.share-bar > span { + display: block; + height: 100%; + background: color-mix(in srgb, var(--danger) 75%, var(--text-main)); +} + +.preprocessing-control-grid { + display: grid; + gap: 12px 18px; + /* auto-fit + 1fr gave a 200px select two 602px tracks. Tracks stop at 300px + and the row stops at its content instead of spanning the panel. */ + grid-template-columns: repeat(auto-fit, minmax(200px, 300px)); + justify-content: start; +} + +.outlier-control-grid { + display: flex; + flex-wrap: wrap; + align-items: flex-end; + gap: 14px 22px; +} + +.outlier-fold-input { + width: 84px; +} + +@media (max-width: 720px) { + .outlier-control-grid { + grid-template-columns: repeat(2, minmax(0, 1fr)); + } +} + +.toggle-switch { + position: relative; + display: inline-block; + width: 44px; + height: 24px; +} + +.toggle-switch input { + opacity: 0; + width: 0; + height: 0; +} + +/* The input is invisible and zero-sized, and styles.css excludes checkboxes from + the studio focus rule, so nothing else can draw the ring. Put it on the slider + (the element the user actually sees) with a solid token-coloured outline. */ +.toggle-switch input:focus-visible + .toggle-slider { + outline: 2px solid var(--primary-color); + outline-offset: 2px; +} + +.toggle-slider { + position: absolute; + cursor: pointer; + top: 0; + left: 0; + right: 0; + bottom: 0; + /* OFF must read as a control and as "not on": the old near-white fill hit 1.2:1. */ + background: color-mix(in srgb, var(--text-muted) 22%, white); + transition: + background 0.22s ease, + border-color 0.22s ease, + box-shadow 0.22s ease; + border-radius: 999px; + border: 1px solid color-mix(in srgb, var(--text-muted) 75%, transparent); +} + +.toggle-slider:before { + content: ""; + position: absolute; + height: 18px; + width: 18px; + left: 3px; + top: 2.5px; + background: var(--card-bg); + border-radius: 50%; + transition: + transform 0.22s var(--ease-standard), + box-shadow 0.22s ease; + /* Ring keeps the knob legible on both the OFF and ON track. */ + box-shadow: 0 2px 6px rgba(15, 23, 42, 0.18), inset 0 0 0 1px rgba(15, 23, 42, 0.12); +} + +.toggle-switch input:checked + .toggle-slider { + /* Flat primary: every other studio gradient is a near-white surface wash, and + a two-stop brand ramp is invisible on a 44x24 track anyway. */ + background: var(--primary-color); + border-color: rgba(43,51,233,0.24); + box-shadow: 0 5px 14px color-mix(in srgb, var(--primary-color) 22%, transparent); +} + +.toggle-switch input:checked + .toggle-slider:before { + transform: translateX(20px); +} + +:host(.outlier-preprocessing-workspace) .preprocessing-detail-panel { + grid-template-columns: minmax(260px, 560px) minmax(0, 1fr); +} + +@media (max-width: 900px) { + :host(.outlier-preprocessing-workspace) .preprocessing-detail-panel { + grid-template-columns: minmax(0, 1fr); + } +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.html b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.html new file mode 100644 index 000000000..6aaf04a7d --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.html @@ -0,0 +1,230 @@ + + +
    +
    + @if (dh.selectedPrepVariable('outlier'); as variable) { + @let rule = dh.outlierRuleFor(variable); + @let state = dh.prepVariableStateLabel('outlier', variable); +
    +
    + Numerical variable +

    {{ dh.variableLabel(variable) }}

    +
    + + {{ state }} + +
    + +
    +
    + + Outlier winsorizer +
    + @if (!rule.enabled) { +

    Turn on clipping for this variable in the list.

    + } +
    +
    + Method +
    + @for (strategy of dh.outlierStrategies; track strategy.value) { + + } +
    +
    +
    + Clip +
    + @for (tail of dh.outlierTails; track tail.value) { + + } +
    +
    + +
    + @if (dh.outlierRuleError(variable)) { +
    {{ dh.outlierRuleError(variable) }}
    + } + @if (dh.outlierBoundsView(variable); as bounds) { +
    +
    + @if (bounds.loLabel) { + {{ bounds.loLabel }} + } + @if (bounds.hiLabel) { + {{ bounds.hiLabel }} + } +
    +
    + + + @if (bounds.cappedLeft !== null) { + + + } + @if (bounds.cappedRight !== null) { + + + } +
    +
    + {{ bounds.min.toLocaleString() }} + {{ dh.distributionMapValue(variable, 'Median') }} + {{ dh.distributionMapValue(variable, 'Max') }} +
    + {{ bounds.note }} +
    + } +
    + + @let distribution = dh.distributionFor(variable); + + + } @else { +
    + +

    No variables match the current search.

    + Clear the search field to choose a numerical variable. +
    + } +
    + +
    + @if (dh.selectedPrepVariable('outlier'); as reportVariable) { +
    +
    Values that would be capped · per dataset
    + +
    + + @if (dh.isLoadingOutlierPreview) { +
    + + + +
    + } + + @if (!dh.isLoadingOutlierPreview && dh.outlierPreviewError) { +
    {{ dh.outlierPreviewError }}
    + } + + @if (!dh.isLoadingOutlierPreview && dh.outlierPreviewRowsFor(reportVariable).length > 0) { +
    + + + + + + + + + + + + + @for (row of dh.outlierPreviewRowsFor(reportVariable); track row.variable + '-' + row.dataset) { + + + + + + + + + } + +
    DatasetLower boundUpper boundBelowAboveShare capped
    {{ row.dataset }}{{ row.lowerBound }}{{ row.upperBound }}{{ row.lowerOutliers }}{{ row.upperOutliers }}
    +
    + } + + @if (dh.coverageFor(reportVariable); as reportCoverage) { +

    Bounds are fitted per dataset. {{ reportCoverage.missing.toLocaleString() }} missing values ({{ dh.coveragePercent(reportCoverage) }}) are skipped.

    + } + } +
    +
    diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.spec.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.spec.ts new file mode 100644 index 000000000..f0555693d --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.spec.ts @@ -0,0 +1,102 @@ +import { Component } from '@angular/core'; +import { TestBed } from '@angular/core/testing'; +import { vi } from 'vitest'; +import type { DataHandlingComponent } from '../../data-handling.component'; +import { DATA_HANDLING } from '../../data-handling.context'; +import type { VariableRow } from '../../data-handling.models'; +import { OutlierStepComponent } from './outlier-step.component'; + +@Component({ + imports: [OutlierStepComponent], + template: `
    `, +}) +class HostComponent {} + +const age = { code: 'age', label: 'Age' } as VariableRow; + +function render({ enabled = true, loPct = 5 as number | null, hiPct = 95 as number | null } = {}) { + const rule = { variableCode: 'age', enabled, strategy: 'iqr', tail: 'both', fold: 1.5 }; + const dh = { + prepSearch: { missing: '', outlier: '', longitudinal: '' }, + prepGroups: () => [{ key: 'not-applied', title: 'Not applied', variables: [age] }], + filteredPrepVariables: () => [age], + isPrepVariableSelected: () => true, + selectPrepVariable: vi.fn(), + batchMenuOpen: () => null, + toggleBatchMenu: vi.fn(), + toggleAllOutliers: vi.fn(), + outlierRuleSummary: () => 'IQR × 1.5', + onOutlierEnabledChange: vi.fn(), + selectedPrepVariable: () => age, + variableLabel: (variable: VariableRow) => variable.label, + prepVariableStateLabel: () => 'Not applied', + prepVariableHasPendingChange: () => false, + outlierRuleFor: vi.fn(() => rule), + outlierStrategies: [ + { value: 'iqr', label: 'IQR' }, + { value: 'zscore', label: 'Z-score' }, + ], + outlierTails: [{ value: 'both', label: 'Both' }], + outlierTailDisplay: (tail: string) => tail, + outlierFoldDisplay: () => 'Fold', + onOutlierStrategyChange: vi.fn(), + onOutlierTailChange: vi.fn(), + onOutlierFoldChange: vi.fn(), + outlierRuleError: () => null, + outlierBoundsView: () => ({ + loLabel: '10', hiLabel: '90', loPct, hiPct, iqrLeft: 20, iqrWidth: 40, medianLeft: 50, + cappedLeft: null, cappedRight: null, min: 0, note: 'note', + }), + distributionMapValue: () => '1', + distributionFor: () => [], + coverageFor: () => null, + coveragePercent: () => '0%', + summaryUnavailableNote: () => 'n/a', + hasOutlierRulesForPreview: true, + isLoadingOutlierPreview: false, + outlierPreviewError: null, + outlierPreviewRowsFor: () => [], + previewOutlierReport: vi.fn(), + barWidth: () => 0, + }; + TestBed.configureTestingModule({ providers: [{ provide: DATA_HANDLING, useValue: dh as unknown as DataHandlingComponent }] }); + const fixture = TestBed.createComponent(HostComponent); + fixture.detectChanges(); + return { dh, fixture, el: fixture.nativeElement as HTMLElement }; +} + +describe('OutlierStepComponent', () => { + it('puts the on/off switch in each rail row without selecting the row', () => { + const { dh, el } = render(); + const toggle = el.querySelector('.preprocessing-variable-btn .toggle-switch input')!; + expect(toggle.checked).toBe(true); + toggle.click(); + expect(dh.onOutlierEnabledChange).toHaveBeenCalledWith(age, false); + expect(dh.selectPrepVariable).not.toHaveBeenCalled(); + expect(el.querySelector('.prep-rail-action')?.getAttribute('data-state')).toBe('set'); + }); + + it('reads the rule once and disables the controls while clipping is off', async () => { + const { dh, fixture, el } = render({ enabled: false }); + expect(el.querySelector('.preprocessing-detail-section > .preprocessing-consequence')?.textContent).toContain('Turn on clipping'); + const methods = Array.from(el.querySelectorAll('[aria-label="Outlier method"] button')); + expect(methods.map((button) => [button.disabled, button.getAttribute('aria-checked')])).toEqual([ + [true, 'true'], + [true, 'false'], + ]); + // once per rail row (switch + action line) and once for the detail panel + expect(dh.outlierRuleFor).toHaveBeenCalledTimes(3); + await fixture.whenStable(); + expect(el.querySelector('.outlier-fold-input')?.disabled).toBe(true); + expect(el.querySelector('.prep-rail-action')?.getAttribute('data-state')).toBe('off'); + }); + + it('keeps the bound labels on the strip near its edges', () => { + const shift = (el: HTMLElement) => Array.from(el.querySelectorAll('.bounds-labels span'), (span) => span.style.transform); + expect(shift(render().el)).toEqual(['translateX(0px)', 'translateX(-100%)']); + TestBed.resetTestingModule(); + expect(shift(render({ loPct: 30, hiPct: 88 }).el)).toEqual(['translateX(-50%)', 'translateX(-50%)']); + TestBed.resetTestingModule(); + expect(shift(render({ loPct: null, hiPct: 50 }).el)).toEqual(['translateX(0px)', 'translateX(-50%)']); + }); +}); diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.ts new file mode 100644 index 000000000..d54fef1fc --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/outlier-step/outlier-step.component.ts @@ -0,0 +1,32 @@ +import { ChangeDetectionStrategy, Component, inject } from '@angular/core'; +import { FormsModule } from '@angular/forms'; +import { DATA_HANDLING } from '../../data-handling.context'; +import { PrepRailSlotDirective, PrepVariableRailComponent } from '../prep-variable-rail/prep-variable-rail.component'; + +/** + * The outlier-capping sub-step workspace: the numerical-variable rail (with a per-row + * on/off switch and the "Batch" menu), the detail panel with the bounds controls, and + * the outlier report. + * Attached to the step's workspace
    , so the DOM and selectors are unchanged. + */ +@Component({ + selector: 'section[app-outlier-step]', + imports: [FormsModule, PrepVariableRailComponent, PrepRailSlotDirective], + templateUrl: './outlier-step.component.html', + styleUrl: './outlier-step.component.css', + // Reads the pipeline's plain fields, so it refreshes with the pipeline (see DATA_HANDLING). + changeDetection: ChangeDetectionStrategy.Eager, +}) +export class OutlierStepComponent { + protected readonly dh = inject(DATA_HANDLING); + + /** + * Keeps a bound's label on the strip: centred on its marker, but pinned to start or + * end there when the marker sits within 12% of an edge. + */ + protected boundLabelShift(pct: number | null): string { + const at = pct ?? 0; + if (at < 12) return 'translateX(0)'; + return at > 88 ? 'translateX(-100%)' : 'translateX(-50%)'; + } +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.css b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.css new file mode 100644 index 000000000..38f4731cd --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.css @@ -0,0 +1,101 @@ +/* The rail's slice of the preprocessing step's styles, in the step's cascade order. */ + +@import '../../styles/search-and-empty-states.shared.css'; + +/* The panel's 1px icon nudge (`.stats-panel-shell i`), repeated for this template. */ +:host-context(.stats-panel-shell) i { + transform: translateY(1px); +} + +/* Preprocessing stations are ONE card, not two. styles.css owns the card, the + rail/panel split and the scroll contract; both summary panels use + .statistics-browser as a left rail, so the pill toolbar that used to live here + made this the only surface in the step that rendered the shared class + differently — the adjacent-panel test in DESIGN.md. This file owns the row + internals only. */ +:host(.preprocessing-browser) .chart-search { + margin-top: 2px; +} + +:host(.preprocessing-browser) .preprocessing-search { + width: 100%; + min-width: 0; +} + +/* A 240px rail has no room for a count justified to the far edge. */ +:host(.preprocessing-browser) .statistics-browser-header { + justify-content: flex-start; + gap: 6px; +} + +:host(.preprocessing-browser) .preprocessing-variable-group { + gap: 2px; + margin-top: 12px; +} + +/* The group heading already names the state, so repeating it on every row said + the same thing up to 9 times per station; the state dot carries it instead. */ +:host(.preprocessing-browser) .preprocessing-state-mini { + display: none; +} + +:host(.preprocessing-browser) .empty-state-block { + margin-top: 8px; +} + +/* The shared .statistics-browser row owns layout, padding and selection. + Only the narrower rail's type size is a local concern. */ +:host(.preprocessing-browser) .statistics-variable-btn > span:first-of-type { + font-size: 0.8rem; +} + +/* The per-pill state chip was removed because it repeated its own group heading + up to 9 times. The state still has to scan at a glance, so it moves to a dot + that inherits from the group. Colours are deepened toward --text-main because + --success (2.5:1) and --accent-color (1.9:1) both fail the 3:1 graphic floor. */ +.preprocessing-variable-btn::before { + content: ''; + flex: 0 0 auto; + width: 7px; + height: 7px; + border-radius: 50%; + background: transparent; + /* 55% hit 2.59:1 against the pill; 75% is 4.2:1 and matches the toggle track border. */ + box-shadow: inset 0 0 0 1.5px color-mix(in srgb, var(--text-muted) 75%, transparent); +} + +/* The boxed row and its hover/active border are gone: the shared rail row is + flat with a leading accent edge, so a per-surface border here was the only + thing still making this rail look different from the summary rails. */ +.preprocessing-variable-group[data-state='applied'] .preprocessing-variable-btn::before { + background: color-mix(in srgb, var(--success) 55%, var(--text-main)); + box-shadow: none; +} + +/* Below the two columns the rail can no longer sit beside the editor; it becomes + a banded footer of the same card instead of a detached box. */ +.preprocessing-variable-group[data-state='pending'] .preprocessing-variable-btn::before { + background: color-mix(in srgb, var(--accent-color) 42%, var(--filter-color)); + box-shadow: none; +} + +.preprocessing-variable-group[data-state='default'] .preprocessing-variable-btn::before { + background: var(--primary-color); + box-shadow: none; +} + +.preprocessing-variable-btn { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + row-gap: 1px; + column-gap: 8px; +} + +.preprocessing-variable-btn::before { + display: none; +} + +.preprocessing-variable-btn .statistics-variable-meta { + align-items: flex-start; + gap: 6px; +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.html b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.html new file mode 100644 index 000000000..4bcd13d6b --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.html @@ -0,0 +1,43 @@ +@let kind = step(); +@let groups = dh.prepGroups(kind); +
    + {{ heading() }} + {{ dh.filteredPrepVariables(kind).length }} shown + +
    +
    + + +
    + +@for (group of groups; track group.key) { +
    +
    {{ group.title }} {{ group.variables.length }}
    + @for (variable of group.variables; track variable.code) { + @let state = dh.prepVariableStateLabel(kind, variable); + + } +
    +} +@if (groups.length === 0) { +
    + +

    No variables match the current search.

    + {{ emptyHint() }} +
    +} diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.spec.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.spec.ts new file mode 100644 index 000000000..5b588f9b9 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.spec.ts @@ -0,0 +1,91 @@ +import { Component } from '@angular/core'; +import { TestBed } from '@angular/core/testing'; +import { vi } from 'vitest'; +import type { DataHandlingComponent } from '../../data-handling.component'; +import { DATA_HANDLING } from '../../data-handling.context'; +import type { PreprocessingGroup, VariableRow } from '../../data-handling.models'; +import { PrepRailSlotDirective, PrepVariableRailComponent } from './prep-variable-rail.component'; + +const age = { code: 'age', label: 'Age' } as VariableRow; +const sex = { code: 'sex', label: 'Sex' } as VariableRow; + +@Component({ + imports: [PrepVariableRailComponent, PrepRailSlotDirective], + template: ` + + `, +}) +class HostComponent { + step: 'missing' | 'longitudinal' = 'missing'; + searchId: string | null = 'preprocessing-search'; + groupStates = true; +} + +function render(groups: PreprocessingGroup[], configure: (host: HostComponent) => void = () => {}) { + const dh = { + prepSearch: { missing: '', outlier: '', longitudinal: '' }, + prepGroups: vi.fn(() => groups), + filteredPrepVariables: () => groups.flatMap((group) => group.variables), + prepVariableStateLabel: (_kind: string, variable: VariableRow) => (variable === age ? 'Pending' : 'Default'), + prepVariableHasPendingChange: (_kind: string, variable: VariableRow) => variable === age, + isPrepVariableSelected: (_kind: string, variable: VariableRow) => variable === sex, + selectPrepVariable: vi.fn(), + variableLabel: (variable: VariableRow) => variable.label, + }; + TestBed.configureTestingModule({ providers: [{ provide: DATA_HANDLING, useValue: dh as unknown as DataHandlingComponent }] }); + const fixture = TestBed.createComponent(HostComponent); + configure(fixture.componentInstance); + fixture.detectChanges(); + return { dh, el: fixture.nativeElement as HTMLElement }; +} + +describe('PrepVariableRailComponent', () => { + const groups: PreprocessingGroup[] = [ + { key: 'pending', title: 'Pending', variables: [age] }, + { key: 'default', title: 'Default (NA removal)', variables: [sex] }, + ]; + + it('renders the heading, count, projected batch menu and search', () => { + const { el } = render(groups); + const header = el.querySelector('.statistics-browser-header')!; + expect(header.querySelector('span')?.textContent).toBe('Variables · action'); + expect(header.querySelector('small')?.textContent).toBe('2 shown'); + expect(header.querySelector('.batch-menu')?.textContent).toBe('batch'); + const search = el.querySelector('input.preprocessing-search')!; + expect(search.id).toBe('preprocessing-search'); + expect(search.getAttribute('aria-label')).toBe('Search preprocessing variables'); + }); + + it('groups the variables by state with the step slots in each row', () => { + const { dh, el } = render(groups); + const rendered = Array.from(el.querySelectorAll('.preprocessing-variable-group')); + expect(rendered.map((group) => group.getAttribute('data-state'))).toEqual(['pending', 'default']); + const [ageRow, sexRow] = Array.from(el.querySelectorAll('.preprocessing-variable-btn')); + expect(ageRow.querySelector('.statistics-variable-meta')?.firstElementChild?.className).toBe('lead'); + expect(ageRow.querySelector('.preprocessing-state-mini')?.className).toContain('state-pending'); + expect(sexRow.querySelector('.preprocessing-state-mini')?.className).toContain('state-default'); + expect(ageRow.lastElementChild?.textContent).toBe('age-action'); + expect(sexRow.classList.contains('active')).toBe(true); + ageRow.click(); + expect(dh.selectPrepVariable).toHaveBeenCalledWith('missing', age); + expect(dh.prepGroups).toHaveBeenCalledWith('missing'); + }); + + it('can leave out the group state and search id, and shows the no-match state', () => { + const { el } = render([], (host) => { + host.step = 'longitudinal'; + host.searchId = null; + host.groupStates = false; + }); + expect(el.querySelector('input.preprocessing-search')?.hasAttribute('id')).toBe(false); + expect(el.querySelector('.empty-state-block span')?.textContent).toBe('Clear the search.'); + TestBed.resetTestingModule(); + const withGroups = render(groups, (host) => (host.groupStates = false)).el; + expect(withGroups.querySelector('.preprocessing-variable-group')?.hasAttribute('data-state')).toBe(false); + }); +}); diff --git a/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.ts b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.ts new file mode 100644 index 000000000..594c60284 --- /dev/null +++ b/src/app/pages/experiment-studio/data-handling/preprocessing-station/prep-variable-rail/prep-variable-rail.component.ts @@ -0,0 +1,67 @@ +import { ChangeDetectionStrategy, Component, Directive, TemplateRef, computed, contentChildren, inject, input } from '@angular/core'; +import { NgTemplateOutlet } from '@angular/common'; +import { FormsModule } from '@angular/forms'; +import { DATA_HANDLING } from '../../data-handling.context'; +import { PrepKind, VariableRow } from '../../data-handling.models'; + +/** Where a step's extra per-variable markup goes in a rail row. */ +export type PrepRailSlotName = + /** In the meta line, before the state chip (missing values: the coverage figure). */ + | 'lead' + /** In the meta line, after the state chip (outliers: the on/off switch). */ + | 'trail' + /** Below the label, across the row (the variable's current action). */ + | 'action'; + +export interface PrepRailSlotContext { + $implicit: VariableRow; +} + +/** + * Marks an `` inside a rail as the step's markup for one slot of every + * row, e.g. `…`. + */ +@Directive({ selector: 'ng-template[prepRailSlot]' }) +export class PrepRailSlotDirective { + readonly name = input.required({ alias: 'prepRailSlot' }); + readonly template = inject>(TemplateRef); + + static ngTemplateContextGuard(_dir: PrepRailSlotDirective, _ctx: unknown): _ctx is PrepRailSlotContext { + return true; + } +} + +/** + * The variable rail shared by the three preprocessing sub-steps: heading with the shown + * count (and the step's batch menu, projected as `.batch-menu`), search, the variables + * grouped by state (pending / applied / default / not applied) and the no-match state. + * Each step adds its own per-row markup through `prepRailSlot` templates. + * Attached to the step's `