Migrate SklearnWrapper to narwhals, add polars support

[solegalli]

Summary

SklearnWrapper (and the deprecated alias SklearnTransformerWrapper, which still raises its FutureWarning) now works with pandas, polars and any other eager dataframe supported by narwhals, and returns the library it receives.

How data goes to and comes back from the wrapped scikit-learn transformer:

• pandas: native path, as before. X[variables] goes to scikit-learn, and in fit the cloned transformer is set with set_output(transform="pandas") (except FunctionTransformer, which warns with it when its function returns an array). Creators (OneHotEncoder, PolynomialFeatures) then get a dataframe back with the input's index, which is joined with nw.concat(how="horizontal"). There is no pd.DataFrame(...)/pd.concat any more, so the module no longer imports pandas.
• polars / other backends: nw_X.select(variables).to_numpy() goes to scikit-learn. If the array is object and the selection has nulls, None becomes np.nan, which is what scikit-learn gets from pandas. Without this, polars nulls in text columns are not imputed by SimpleImputer, and OrdinalEncoder/OneHotEncoder treat them as a category (cat_None). The output is rebuilt with nw.from_numpy(..., backend=nw_X.implementation), and NaN turns back into null (float arrays, only when NaN is present) so missing values stay missing for later polars steps. FunctionTransformer gets the native polars frame, because the user's function is written for their dataframe. If it returns a dataframe, its columns are taken by position, like pandas X[cols] = df does.
• selectors: one column selection of the kept features (pandas X[kept], narwhals select) instead of reorder, then drop.
• The explicit set_output also protects the inner transformer from a global sklearn.set_config(transform_output=...).

Init: issubclass(transformer.__class__, ...) became isinstance(transformer, TransformerMixin). The TypeError message now ends Got {transformer} instead. (it was got ... instead. ). I removed the unreachable "transformer has no inverse_transform" branch: every entry in _INVERSE_TRANSFORM has the method.

Benchmarks

Machine: Apple silicon, Python 3.14, pandas 3.0.3, polars 1.43, narwhals 2.24, scikit-learn 1.9. Times are in ms, median of 7 repeats; "before" is the last working pandas version (ec26810^, before check_X returned narwhals). Before and after ran in alternating processes, 3 times each, and the table shows the median of those. The machine was shared with other jobs, so differences under about 1 ms, or about 10% at 500k rows, are noise.

Candidate implementations (transform only)

pandas, 500k rows (the size that drives the decision). "current" is the old code; "set_output" is what this PR uses; "narwhals" is nw.from_numpy + with_columns / nw.concat:

case	cols	current	set_output (chosen)	narwhals
StandardScaler (all num)	6	3.0	2.3	3.0
StandardScaler (all num)	21	24.1	18.1	20.7
PolynomialFeatures (4 vars)	6	75.6	36.1	71.7
PolynomialFeatures (4 vars)	21	61.3	31.8	78.9
OneHotEncoder (1 var, 10 levels)	6	71.9	50.8	65.3
OneHotEncoder (1 var, 10 levels)	21	69.0	33.5	52.0
VarianceThreshold	21	2.2 (reorder + drop)	1.5 (one select)	1.6

A hand-built pd.DataFrame(arr, index=X.index, copy=False) performed about the same as set_output. I chose set_output because it needs no pandas namespace and also protects against the global config. At 10k and 100k rows all variants are within 1 ms of each other.

polars, 500k to 2M rows. The input goes to scikit-learn as native polars (native_in) or as numpy (numpy_in), and the output is built with nw.from_numpy, nw.new_series, set_output("polars") or native pl.from_numpy + with_columns/hstack:

case	rows	cols	native_in + nw.from_numpy	numpy_in + nw.from_numpy	native_in + nw.new_series	set_output(polars)	polars native
StandardScaler	500k	6	4.6	3.5	4.5	4.7	4.8
StandardScaler	500k	21	27.2	21.8	26.2	25.4	26.5
StandardScaler	1M	21	47.3	43.7	48.9	52.3	57.1
Poly (4 vars)	500k	6	42.8	35.0	35.7	38.1	33.3
Poly (4 vars)	1M	21	92.4	85.6	79.0	87.8	92.8
Poly (4 vars)	2M	21	134.4	133.3	158.3	262.1	287.4
OHE	500k	21	50.6	50.3	51.0	50.9	49.8
OHE	1M	21	100.7	99.5	109.2	108.5	106.9

fit with native vs numpy input was the same at every size (e.g. StandardScaler 1M/21: 85.5 vs 94.1, 2M/21: 224.5 vs 222.8). numpy input wins or ties everywhere, and it is the only option that gives pandas-equal results for nulls in text columns, so the choice is numpy input + nw.from_numpy. Turning NaN back into null costs about 1 ms for the np.isnan(...).any() check when there are no NaN (500k x 15), and about 9 ms more at 500k x 15 when there are.

inverse_transform on pandas (500k rows): shallow copy(deep=False) + column assignment 2.2 ms (6 cols) / 8.2 ms (21 cols), deep copy() 7.0 / 15.8, narwhals with_columns 3.2 / 7.5. I chose the shallow copy. Assigning whole columns replaces them in the copy and leaves the user's arrays alone, and a test checks that the input is unchanged.

Whole fit / transform, pandas before vs after (ms)

rows	cols	transformer	fit before	fit after	transform before	transform after	inverse before	inverse after
10,000	6	StandardScaler	0.7	0.7	0.6	0.7	0.4	0.4
10,000	6	Poly(4 vars)	0.5	0.5	1.0	0.9	-	-
10,000	6	OHE	0.5	0.4	1.4	1.1	-	-
10,000	21	StandardScaler	0.8	1.0	1.0	1.2	0.7	0.7
10,000	21	VarianceThreshold	1.0	0.9	0.2	0.1	-	-
100,000	6	Poly(4 vars)	0.8	0.7	6.8	5.2	-	-
100,000	6	OHE	1.4	1.5	7.9	6.3	-	-
100,000	21	StandardScaler	4.3	4.1	3.4	2.8	3.1	3.6
100,000	21	Poly(4 vars)	0.7	0.7	5.8	4.6	-	-
100,000	21	OHE	1.4	1.8	7.5	7.7	-	-
500,000	6	StandardScaler	5.2	5.2	3.3	3.2	3.6	4.2
500,000	6	Poly(4 vars)	1.1	1.3	36.8	21.8	-	-
500,000	6	OHE	4.9	6.3	35.7	31.6	-	-
500,000	21	StandardScaler	18.5	17.7	12.0	10.8	15.6	14.5
500,000	21	Poly(4 vars)	1.2	1.2	36.9	19.3	-	-
500,000	21	OHE	4.8	6.3	38.1	30.5	-	-
500,000	21	VarianceThreshold	27.9	22.9	1.7	1.8	-	-

"inverse before" did no copy; that was the in-place bug fixed below. fit is the scikit-learn fit plus under 0.2 ms: at 500k x 21, wrapper fit took 5.2 ms and scikit-learn's own OneHotEncoder fit took 5.1 ms. So the fit differences in the table are noise.

Whole fit / transform, polars (ms)

rows	cols	transformer	fit	transform	inverse
500,000	21	StandardScaler	25.0	18.9	10.4
500,000	21	Poly(4 vars)	2.4	45.2	-
500,000	21	OHE	21.8	62.7	-
500,000	21	VarianceThreshold	31.7	0.2	-
1,000,000	21	StandardScaler	75.3	40.1	27.3
1,000,000	21	Poly(4 vars)	4.3	80.6	-
1,000,000	21	OHE	38.2	111.5	-
2,000,000	21	StandardScaler	95.6	65.7	43.0
2,000,000	21	Poly(4 vars)	7.0	160.3	-
2,000,000	21	OHE	70.4	199.7	-

At the same size, polars creators are slower than pandas. There are two causes, and neither is in the wrapper's own logic. First, scikit-learn returns a row-major array: pandas wraps it without copying, while polars has to copy it into columns (from_numpy of 500k x 15 floats costs about 20 ms with every constructor tried). Second, a polars text column must be converted to a numpy object array (about 9 ms per 500k rows), and scikit-learn does that same conversion when it gets polars directly.

Behaviour

pandas: identical to the last working version (ec26810^), checked on 94 recorded cases. The cases cover all 16 supported transformers, 7 creator and 4 selector configurations, variables=None and a list, reordered columns at transform, a custom index, NaN in numeric and text columns, integer column names, category dtype, return_empty, error paths and a global transform_output config. Compared: columns, dtypes, index, values, warnings, variables_, features_to_drop_, get_feature_names_out and inverse_transform. The only difference is the bug fix below.

polars: same values as pandas in every case, including missing values in text columns (imputed, cat_nan column, NaN kept by OrdinalEncoder). Differences that come from the backend:

• missing values come back as null, not NaN.
• transformer_.feature_names_in_ is not set for polars, because scikit-learn gets numpy. As a result, get_feature_names_out(input_features) for OneHotEncoder/PolynomialFeatures accepts any list of the right length on polars. On pandas, scikit-learn requires input_features to equal the fitted variables. Valid calls give the same names on both backends.
• a selector that drops every column gives a (0, 0) polars frame, because polars cannot have rows without columns (pandas keeps the index).
• FunctionTransformer's function gets the polars frame, so its result follows polars rules. For example, lambda x: x + 1 on an int and a float column gives two floats in polars.

Bugs fixed

• inverse_transform modified the user's dataframe in place (X[variables] = ... on the input, because check_X does not copy). It now works on a shallow copy (pandas) or a new frame (narwhals). test_inverse_transform checks that the input is unchanged and fails without the fix.

Tests

• tests/test_wrappers/test_sklearn_wrapper.py is rewritten to the conventions: # init parameters (one test per error message, with wrong values and types, test_init_param_assignment, the deprecation warning with its full message), then # fit and transform. It uses make_df for both backends, plain dict data with explicit expected values, frame_to_dict / null_count / make_series, and one list/array target test. The pandas-only tests cover integer column names, the index and the category dtype. It no longer downloads the California housing dataset.
• In test_check_estimator_wrappers.py, test_return_empty moved (as two tests, on both backends) to the main file, and the remaining pytest.raises got a full match=.

Failing tests (tests/test_wrappers, the only folder that imports the wrappers):

	failed	passed
base narwhals-migration	64	14
this PR	1	135

The remaining failure, test_sklearn_transformer_wrapper, fails on the base and also on ec26810^. scikit-learn's check_fit_score_takes_y passes numpy arrays, and check_X rejects them; imputers and outliers have the same failure. flake8 feature_engine tests is clean. mypy feature_engine reports the same 2 errors as the base (in datetime_subtraction.py and log.py).

Docs

• Docstring: types say "dataframe", with a polars example (run, real output). The three pandas examples still match.
• User guide: added a "With polars" section with a run example that shows nulls being imputed and kept. I fixed the existing examples that no longer ran or matched:
  • OneHotEncoder used sparse=False, which scikit-learn >= 1.7 rejects; it now uses sparse_output=False.
  • data['embarked'].fillna('C', inplace=True) does nothing under pandas copy-on-write; it is now an assignment.
  • The printed titanic output now shows the cabin values the loader returns under pandas 3 (NaN).
  • The SelectKBest example selected columns with dtypes != 'O', which picks text columns on pandas 3 and fails. It now uses find_numerical_variables.
  • In the pipeline output, row 10's cabin value is 0, not 2. It was already wrong before this PR.

Needs decision

1. Turning NaN back into null for polars. I did this so a later fill_null-based feature-engine step (e.g. MeanImputer) still sees the missing values. scikit-learn's own set_output("polars") returns NaN instead. It costs about 1 ms per 500k x 15 without NaN, and about 9 ms with NaN. Keep it, or return NaN like scikit-learn?
2. None becomes NaN before scikit-learn for polars text columns. This gives pandas-equal results. Without it, scikit-learn on polars leaves nulls in text columns unimputed and encodes them as a None category. Keep it?
3. TypeError message changed from got {transformer} instead. to Got {transformer} instead. to follow the convention.

Pre-existing issues, not fixed

• OneHotEncoder and PolynomialFeatures fail with pandas integer column names. scikit-learn's get_feature_names_out builds the new names by joining strings, so this fails before and after this PR.
• OrdinalEncoder inverse_transform fails on pandas 3 when the encoded column has NaN: ValueError: assignment destination is read-only. scikit-learn writes into the read-only array that pandas copy-on-write hands out, and calling scikit-learn directly fails the same way.
• check_estimator fails on check_fit_score_takes_y, because it uses numpy input (see Tests).
• if getattr(transformer, "sparse", False) in __init__ is dead code with scikit-learn >= 1.7, which has no sparse parameter. Left as is.