(With Claude): The null-filling path for schema evolution exists but is unreachable, and the storage reader does not implement it at all.
1. The set is always empty. TableSet::set_nullable and set_column_default (crates/query/src/plan/table.rs:39,44) are the only ways to populate column_defaults, and neither is called anywhere in the repo:
$ git grep -n "set_nullable\|set_column_default" -- crates/
crates/query/src/plan/table.rs:39: pub fn set_column_default(...)
crates/query/src/plan/table.rs:44: pub fn set_nullable(...)
So default_null_columns() always returns an empty HashSet, and the branch that would inject NullArrays in crates/query/src/scan/parquet/file.rs:170 is dead code. Every projected column missing from a chunk schema takes the else arm instead:
tracing::error!("column ...is not found in...");
anyhow::bail!(ColumnDoesNotExist::new(self.table_name.to_string(), name));
2. The storage reader ignores the parameter. impl TableReader for SnapshotTableReader takes it as _default_null_columns (crates/query/src/scan/storage/reader.rs:23) and never uses it, so populating the set would fix the parquet path only.
Effect. Adding a column to a chunk builder is not backward compatible: selecting the new field over any chunk written before the change returns ColumnDoesNotExist rather than nulls. A concrete instance is the solana transaction_config column added in 8de9324 with query support in aa62e79 — selecting transactionConfig over chunks predating 8de9324 errors. The same applies to any future column addition on any dataset kind.
Suggested fix: declare newly added columns via set_nullable in the per-dataset query definitions, and honour default_null_columns in SnapshotTableReader::read so both scan paths behave the same. Worth deciding whether the default should instead be to null-fill any missing projected column, since the current behaviour makes every schema addition a breaking change for historical ranges.
(With Claude): The null-filling path for schema evolution exists but is unreachable, and the storage reader does not implement it at all.
1. The set is always empty.
TableSet::set_nullableandset_column_default(crates/query/src/plan/table.rs:39,44) are the only ways to populatecolumn_defaults, and neither is called anywhere in the repo:So
default_null_columns()always returns an emptyHashSet, and the branch that would injectNullArrays incrates/query/src/scan/parquet/file.rs:170is dead code. Every projected column missing from a chunk schema takes theelsearm instead:2. The storage reader ignores the parameter.
impl TableReader for SnapshotTableReadertakes it as_default_null_columns(crates/query/src/scan/storage/reader.rs:23) and never uses it, so populating the set would fix the parquet path only.Effect. Adding a column to a chunk builder is not backward compatible: selecting the new field over any chunk written before the change returns
ColumnDoesNotExistrather than nulls. A concrete instance is the solanatransaction_configcolumn added in 8de9324 with query support in aa62e79 — selectingtransactionConfigover chunks predating 8de9324 errors. The same applies to any future column addition on any dataset kind.Suggested fix: declare newly added columns via
set_nullablein the per-dataset query definitions, and honourdefault_null_columnsinSnapshotTableReader::readso both scan paths behave the same. Worth deciding whether the default should instead be to null-fill any missing projected column, since the current behaviour makes every schema addition a breaking change for historical ranges.