Repository navigation
Register each function under the name Spark and Flink take where they own the PostgreSQL one - #174
Merged
estebanzimanyi merged 1 commit intoOct 11, 2026
Conversation
… own the PostgreSQL one The catalog states on each SQL signature whose PostgreSQL name Spark or Flink defines or reserves the name these engines publish it under, altSqlName: floatRound for round(tfloat, integer), spanLower for lower(intspan), temporalHash for hash(tint), setUnnest for unnest(intset). The Spark UDF surface, the typed Spark SQL surface and the Flink SQL surface register every function under that name where the catalog states one, else under its SQL name, so none of them registers round, lower, length, abs or another name the engine owns, which on Spark replaces the engine's own function for the rest of the Spark session, and which Flink cannot reach or refuses to parse. codegen_jvm.py takes it in #signatures, which both SQL engines read, and codegen_spark_udfs.py in #published_name, read wherever the UDF surface groups, matches and ranks functions by name. A bare aggregate the catalog also states under its Agg name is carried under that name alone: merge, whose scalar is temporalMerge, is registered as mergeAgg, never as merge, as the portable naming plan decides for both engines. tools/check_owned_names.py locks it in: a CI step generates the UDF and the two SQL surfaces and fails on a registered name all of whose catalog signatures carry an altSqlName, a name some signature states without one, as scale over the number and time extents, staying registrable. Measured over MobilityDB master 6882458ebd, the catalog of MEOS-API e384b86941 derived from the installed headers of its libmeos, and the JMEOS jar built on them, generating each surface with main's generator and with this one: check_owned_names.py finds 33 names the engines own on each of main's surfaces (abs, affine, ceil, hash, insert, lower, merge, round, set, unnest, ...) and none on this one's. No other name of main's surfaces is lost, and every name added is an altSqlName of the catalog, 87 on the UDF surface and 98 on each SQL surface, mergeAgg the aggregate. mvn -pl codegen test passes, and the Spark gaps step reports 713 functions, none new. Why: the name a binding publishes is the one the catalog states for its engine, so a query moves between Spark and Flink unchanged and neither engine loses a function of its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The catalog states on each SQL signature whose PostgreSQL name Spark or Flink defines or reserves
the name these engines publish it under, altSqlName: floatRound for round(tfloat,
integer), spanLower for lower(intspan), temporalHash for hash(tint), setUnnest for unnest(intset).
The Spark UDF surface, the typed Spark SQL surface and the Flink SQL surface register every
function under that name where the catalog states one, else under its SQL name, so none of them
registers round, lower, length, abs or another name the engine owns, which on Spark replaces the
engine's own function for the rest of the Spark session, and which Flink cannot reach or refuses
to parse.
codegen_jvm.py takes it in #signatures, which both SQL engines read, and codegen_spark_udfs.py in
#published_name, read wherever the UDF surface groups, matches and ranks functions by name.
A bare aggregate the catalog also states under its Agg name is carried under that name alone:
merge, whose scalar is temporalMerge, is registered as mergeAgg, never as merge, as the
portable naming plan decides for both engines.
tools/check_owned_names.py locks it in: a CI step generates the UDF and the two SQL surfaces and
fails on a registered name all of whose catalog signatures carry an altSqlName, a name some
signature states without one, as scale over the number and time extents, staying registrable.
Measured over MobilityDB master 6882458ebd, the catalog of MEOS-API e384b86941 derived from the
installed headers of its libmeos, and the JMEOS jar built on them, generating each surface with
main's generator and with this one: check_owned_names.py finds 33 names the engines own on each
of main's surfaces (abs, affine, ceil, hash, insert, lower, merge, round, set, unnest, ...) and
none on this one's. No other name of main's surfaces is lost, and every name added is an
altSqlName of the catalog, 87 on the UDF surface and 98 on each SQL surface, mergeAgg the
aggregate. mvn -pl codegen test passes, and the Spark gaps step reports 713 functions, none new.
Why: the name a binding publishes is the one the catalog states for its engine, so a query moves
between Spark and Flink unchanged and neither engine loses a function of its own.