[Discussion] Define python UDF using `CREATE FUNCTION` SQL statement

[milenkovicm]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I would like to be able to define a python function using SQL CREATE FUNCTION statement.

Something like

let sql = r#"
CREATE FUNCTION to_miles(DOUBLE)
RETURNS DOUBLE
LANGUAGE PYARROW
AS '
import pyarrow.compute as pc

conversation_rate_multiplier = 0.62137119

def to_miles(km_data):
    return pc.multiply(km_data, conversation_rate_multiplier)
'
"#;

ctx.sql(sql).await?.show().await?;

ctx.sql("select double_col, to_miles(double_col) from t")
    .await?
    .show()
    .await?;

Describe the solution you'd like

datafusion provides pluggable datafusion::execution::context::FunctionFactory which could be wired to interpret python function definitions.

we could provide column based function definitions (LANGUAGE PYARROW) or row based (LANGUAGE PYTHON)

Describe alternatives you've considered

Additional context

• databricks has similar functionality