From ba7714da35bc2d16a48c4c70f16a824d86385752 Mon Sep 17 00:00:00 2001 From: Ajayjayendran Arulraj Date: Tue, 27 Oct 2026 18:43:05 +0530 Subject: [PATCH] Update CHANGELOG.md --- CHANGELOG.md | 21 --------------------- 1 file changed, 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f5f787..79a9f36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,23 +43,6 @@ - **Dependency upgrades for Snyk findings.** Bumps `fast-uri` and `brace-expansion` to patched versions that address reported URI-parsing, ReDoS, and uncontrolled-recursion vulnerabilities. -### Python models - -- **Project defaults for python-model skills.** Commit `.agents/project/skills//project-defaults.md` in your dbt project so `dj-create-python-model`, `dj-review-python-model`, and `dj-migrate-notebook-to-pymodel` pick up your catalog, schema, authoring mode, and migration rules in Step 0 — without forking DJ skill files. Example templates live in the DJ repo under `templates/_agents-project/skills/` (not deployed). DJ never overwrites `.agents/project/`. -- **Declare upstream lineage in `.python.json`.** Add `"upstream_sources": [{ "type": "trino"|"external", "value": "..." }]` on any Python model; DJ auto-suggests trino entries from `depends_on` on JSON sync when omitted. After each successful Airflow or KPO run, DJ writes `python_model_name` and the structured `python_model_upstream_sources` property to the output Iceberg table — no manual `ALTER TABLE` in your `.python.py`. -- **Model Lineage walks the full Python chain.** When a downstream dbt model sources a Python output table, Data Explorer recursively reads each upstream table's Iceberg properties until it reaches roots (external sources or tables with no further upstream), so a chain like D → C → B, A renders in one view. -- **Cleaner lineage for Python models.** Python models now show their real upstream sources in the lineage graph, without the extra placeholder node that used to appear. -- **Model Lineage distinguishes external vs Trino upstream nodes.** External sources (`s3::`, API, Jenkins) and Trino table refs render with distinct badges in the lineage graph. -- **One Airflow task per KPO Python model.** Models with `compute: kpo` appear as a single task named after the model (e.g. `kpo_infra`), not separate gate/KPO steps — DJ runs gating and the pod inside that task. -- **KPO runs write Iceberg lineage properties.** Post-run `python_model_*` table properties (including `python_model_upstream_sources`) are applied after successful KPO tasks, not only MWAA worker runs. -- **`_trino_io` pandas helpers for python-path models.** Use `read_partition_to_dataframe` and `overwrite_partition_from_dataframe` when you choose a pandas orchestrator; existing SQL helpers are unchanged. - -### Create Python Model - -- **Group options match your project.** The Create Python Model form now lists the groups from the project you've selected, and refreshes when you switch projects. -- **Fix: creating a Python model works reliably.** Creating a Python model and loading its DAG and group options now work as expected. -- **Faster, clearer DAG selection.** Select DAG shows a loading state while options load; **`source_etl`** is omitted from the list because DJ manages that DAG via its generated template; creating a model no longer prompts to modify the selected DAG — task wiring is applied automatically. - ## 2.3.0 ### Framework @@ -76,10 +59,6 @@ - **Fix: Model Lineage opens the correct view on first use.** Opening Model Lineage from a model `.sql`, `.model.json`, or `.yml` file (context menu, editor title, or `Cmd+Shift+L`) now shows the model lineage graph in Data Explorer instead of Column Lineage the first time the panel opens. -### Python models - -- **KPO compute is the default for new Python models.** Create Python Model lets you pick **Compute** (`kpo` or legacy `airflow` worker) and, for KPO, a **Pod size** (`small` through `xlarge`) with vCPU, memory, and workload guidance from `dags/_ext_/kpo_sizes.yml`. DJ ships `kpo_factory.py` and that YAML alongside `etl_helper.py`; set Airflow Variables `airflow_runner_cfg` (image/cluster) and `python_models_s3_bucket` (KPO S3 sync). Scaffolded `.python.py` files include `if __name__ == "__main__"`; Airflow and KPO both execute the script as `__main__` (not a fixed `run_etl` import). Until your KPO pipeline is ready, set `"compute": "airflow"` on existing models. - ### Agent skills - **New `dj-migrate-notebook-to-pymodel`, `dj-verify-pymodel-parity`, and `dj-document-pymodels` skills.** Migrate a legacy Jupyter notebook into a python model with migration plan, generate Trino SQL to verify a python model's output table matches a legacy table, and generate/refresh a topic-level README for a `python_models///` folder — rounding out the python model lifecycle alongside `dj-create-python-model` and `dj-review-python-model`.