Skip to content

Register each function under the name Spark and Flink take where they own the PostgreSQL one - #174

Merged
estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:codegen/alt-sql-name
Oct 11, 2026
Merged

estebanzimanyi merged 1 commit into
MobilityDB:mainfrom
estebanzimanyi:codegen/alt-sql-name

Conversation

@estebanzimanyi

Copy link
Copy Markdown
Member

The catalog states on each SQL signature whose PostgreSQL name Spark or Flink defines or reserves
the name these engines publish it under, altSqlName: floatRound for round(tfloat,
integer), spanLower for lower(intspan), temporalHash for hash(tint), setUnnest for unnest(intset).
The Spark UDF surface, the typed Spark SQL surface and the Flink SQL surface register every
function under that name where the catalog states one, else under its SQL name, so none of them
registers round, lower, length, abs or another name the engine owns, which on Spark replaces the
engine's own function for the rest of the Spark session, and which Flink cannot reach or refuses
to parse.
codegen_jvm.py takes it in #signatures, which both SQL engines read, and codegen_spark_udfs.py in
#published_name, read wherever the UDF surface groups, matches and ranks functions by name.

A bare aggregate the catalog also states under its Agg name is carried under that name alone:
merge, whose scalar is temporalMerge, is registered as mergeAgg, never as merge, as the
portable naming plan decides for both engines.

tools/check_owned_names.py locks it in: a CI step generates the UDF and the two SQL surfaces and
fails on a registered name all of whose catalog signatures carry an altSqlName, a name some
signature states without one, as scale over the number and time extents, staying registrable.

Measured over MobilityDB master 6882458ebd, the catalog of MEOS-API e384b86941 derived from the
installed headers of its libmeos, and the JMEOS jar built on them, generating each surface with
main's generator and with this one: check_owned_names.py finds 33 names the engines own on each
of main's surfaces (abs, affine, ceil, hash, insert, lower, merge, round, set, unnest, ...) and
none on this one's. No other name of main's surfaces is lost, and every name added is an
altSqlName of the catalog, 87 on the UDF surface and 98 on each SQL surface, mergeAgg the
aggregate. mvn -pl codegen test passes, and the Spark gaps step reports 713 functions, none new.

Why: the name a binding publishes is the one the catalog states for its engine, so a query moves
between Spark and Flink unchanged and neither engine loses a function of its own.

… own the PostgreSQL one

The catalog states on each SQL signature whose PostgreSQL name Spark or Flink defines or reserves
the name these engines publish it under, altSqlName: floatRound for round(tfloat,
integer), spanLower for lower(intspan), temporalHash for hash(tint), setUnnest for unnest(intset).
The Spark UDF surface, the typed Spark SQL surface and the Flink SQL surface register every
function under that name where the catalog states one, else under its SQL name, so none of them
registers round, lower, length, abs or another name the engine owns, which on Spark replaces the
engine's own function for the rest of the Spark session, and which Flink cannot reach or refuses
to parse.
codegen_jvm.py takes it in #signatures, which both SQL engines read, and codegen_spark_udfs.py in
#published_name, read wherever the UDF surface groups, matches and ranks functions by name.

A bare aggregate the catalog also states under its Agg name is carried under that name alone:
merge, whose scalar is temporalMerge, is registered as mergeAgg, never as merge, as the
portable naming plan decides for both engines.

tools/check_owned_names.py locks it in: a CI step generates the UDF and the two SQL surfaces and
fails on a registered name all of whose catalog signatures carry an altSqlName, a name some
signature states without one, as scale over the number and time extents, staying registrable.

Measured over MobilityDB master 6882458ebd, the catalog of MEOS-API e384b86941 derived from the
installed headers of its libmeos, and the JMEOS jar built on them, generating each surface with
main's generator and with this one: check_owned_names.py finds 33 names the engines own on each
of main's surfaces (abs, affine, ceil, hash, insert, lower, merge, round, set, unnest, ...) and
none on this one's. No other name of main's surfaces is lost, and every name added is an
altSqlName of the catalog, 87 on the UDF surface and 98 on each SQL surface, mergeAgg the
aggregate. mvn -pl codegen test passes, and the Spark gaps step reports 713 functions, none new.

Why: the name a binding publishes is the one the catalog states for its engine, so a query moves
between Spark and Flink unchanged and neither engine loses a function of its own.
@estebanzimanyi
estebanzimanyi merged commit e8c65dc into MobilityDB:main Oct 11, 2026
2 checks passed
@estebanzimanyi
estebanzimanyi deleted the codegen/alt-sql-name branch October 11, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant