[psaesha]

Pull Request

Description

Adds the Source identifier dataset metadata item. CivicDataLab's scrapers fill it in automatically, naming the scraper and dataset that produced it, e.g. dataspace-scrapers/udise-plus/enrolment. The scrapers read it back to recognise datasets they've already uploaded. That way a dataset is never created twice, and the scrapers' upload records can be rebuilt from DataSpace if they're lost. People never need to fill it in. Editing or clearing it can make a scraper create a duplicate.

The item also gets filled by platform import with no extra code: the import already fills any dataset item it recognises as a source identifier, so new imports from Kaggle, Hugging Face and GitHub will store the platform's own ID here (e.g. owner/dataset-slug). Only values starting with dataspace-scrapers/ belong to a scraper.

Type of Change

• New feature (non-breaking change which adds functionality)

Changes Made

• api/migrations/0054_add_source_identifier_metadata.py: a data migration that creates the item (dataset, text, optional, switched on) if it is missing, and switches it back on if an admin switched it off. Reversing it deletes nothing, since datasets may already carry values for it.
• tests/test_source_identifier_migration.py: covers the item being missing, being switched off, the migration running twice, and an item with the same label on resources being left alone.

Testing

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes (targeted run: pytest tests/test_source_identifier_migration.py, 4 passed)

Test Instructions

1. python manage.py migrate
2. In Django admin, open Metadata and check there is one Source identifier item for datasets, switched on.
3. Open a dataset's edit form: the field appears under "Additional Information".

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have added tests that prove my fix is effective or that my feature works
• I have run black ., isort ., and flake8 and fixed any issues (flake8 at the repo's 100-character line length)
• Database migrations are included (if applicable)

Additional Notes

• The label must stay exactly Source identifier. The scrapers look the item up by that label, on datasets, switched on, and expect only one.
• The item shows on the dataset edit form and public dataset pages, like Source Website. The edit form shows every switched-on dataset item whatever its type, so marking it ADVANCED would not hide it. Hiding it would need a frontend change.
• Migration number: 0054 follows 0053, the newest on dev. The s3_migration branch also has a 0054_file_field_storages.py; whichever merges second needs renumbering or a merge migration.
• Unrelated, not fixed here: the "required value" check in BaseMetadata._validate_data_type compares with is instead of ==, so it never fires and REQUIRED items like Source Website aren't enforced.

🤖 Generated with Claude Code

[anantjain341]

@psaesha merged and deployed to dev. The Source identifier field is live and enabled, so the scrapers can use it now. Thanks!