Exchange content provider by eilmiv · Pull Request #1388 · ElixirTeSS/TeSS

eilmiv · 2026-10-20T15:52:43Z

Summary of changes

Here is an overview of the pull request as a goal tree:

Added exchange of content providers

Export metadata
- Provider HTML page metadata
  - Include keywords in metadata of content provider page
  - Some small open graph metadata fixes
- RDF metadata
  - Events now export provider in Bioschemas RDF
  - Add subjectOf that points to page in TeSS to provider
Import metadata
- Update ingestion process (only Portal providers ingest content providers)
  - Add additional provider metadata download step
    - Add source attribute to ingestor - this was originally only available to write()
    - Introduce shared bioschemas extraction method that is used by OAI-PMH, Bioschemas and RSS ingestors
    - Define two providers to be the same when they have the same URL (this also updates existing check_exists() for content providers)
  - Add additional provider update step
- Implement exchange related features for content providers
  - Provider field locking
    - Add autolocking
  - Provider ingestion metadata: mark automatically ingested records and keep track of last ingestion as well as statistics
  - Preview for content providers in ingestor testing

Motivation and context

Could be seen as the final necessary feature for #1191 and is relevant as the exchange part of the mTeSS-x project.

Screenshots

Checklist

I have read and followed the CONTRIBUTING guide.
I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree to license it to the TeSS codebase under the BSD license.

Conflict resolutions: - app/assets/stylesheets/application.scss: kept master's split stylesheet structure and dropped the inline block that master moved into the base/, components/ and pages/ partials. - app/assets/stylesheets/base/_utilities.scss: relocated the branch's .bulk-import-row .provider-logo rule from that removed inline block so the source test resource logo keeps its sizing. - app/views/sources/_test_results.html.erb: combined master's per-resource error handling with the branch's content_provider_id merge for non-provider test resources. - db/schema.rb: kept version 2026_10_07_141523, the newer of the two, and confirmed the file matches a schema generated by migrating a fresh database. - test/models/event_test.rb: kept both new test groups (provider subjectOf, plus oai_dc and rdf serialization). - test/unit/ingestors/ingestor_test.rb: migrated master's new source responsibility tests to the branch API by assigning ingestor.source and calling write(user). Validation: - bundle exec rails test --exclude '/test_logs_backtrace_on_error/' => 1635 tests, 9136 assertions, 0 failures, 0 errors, 10 skips - All five theme stylesheets compile through the asset pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Provider previews can persist unauthorized changes, and provider reassignment can duplicate existing imports.

3 open issues

What changed in this PR

Adds content-provider exchange to TeSS, preserving provider metadata alongside imported events and materials for the exchange workflow described in #1191.

Changes:

Exports provider links and metadata, and adds portal-only provider ingestion.
Adds provider field locking, ingestion tracking, and previews.
Updates ingestion interfaces and expands regression coverage.

File	Description
test/unit/ingestors/zenodo_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/tess_event_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/taxila/wur_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/vu_material_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/uva_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/uu_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/utwente_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/uhasselt_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/tdcc_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/surf_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/sense_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/rug_ingestor_test.rb	Updates source setup across runs.
test/unit/ingestors/taxila/rst_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/rdnl_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/oscm_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/osci_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/oscd_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/odissei_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/nwo_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/maastricht_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/leiden_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/lcrdm_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/han_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/dtls_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/dcc_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/dans_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/citizen_science_ingestor_test.rb	Updates event/material ingestion setup.
test/unit/ingestors/taxila/4tu_willma_llm_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/taxila/4tu_gpt_llm_ingestor_test.rb	Updates ingestion setup.
test/unit/ingestors/scraper_test.rb	Tests provider counts and repeated imports.
test/unit/ingestors/provider_ingestion_test.rb	Covers provider resolution, updates, and locks.
test/unit/ingestors/oai_pmh_ingestor_test.rb	Tests provider metadata extraction.
test/unit/ingestors/material_rss_ingestor_test.rb	Tests deferred provider resolution.
test/unit/ingestors/material_csv_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/libcal_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/ingestor_test.rb	Tests source context and revised interfaces.
test/unit/ingestors/indico_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/ical_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/heptraining/gray_scott_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/github_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/eventbrite_ingestor_test.rb	Supplies token through source context.
test/unit/ingestors/event_rss_ingestor_test.rb	Tests deferred provider resolution.
test/unit/ingestors/event_csv_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/carpentries_ingestor_test.rb	Uses source-aware ingestion.
test/unit/ingestors/bioschemas_ingestor_test.rb	Tests extraction and provider-reference preservation.
test/models/source_test.rb	Covers provider statistics in logging.
test/models/material_test.rb	Tests exported provider-page links.
test/models/field_lock_test.rb	Tests provider locks and configuration.
test/models/event_test.rb	Tests exported provider-page links.
test/models/content_provider_test.rb	Tests instance-based provider lookup.
test/integration/content_provider_exchange_test.rb	Tests metadata exchange round trips.
test/helpers/application_helper_test.rb	Tests preview images and freshness indicators.
test/controllers/sources_controller_test.rb	Tests provider counts and preview results.
test/controllers/materials_controller_test.rb	Tests autolocking and provider previews.
test/controllers/events_controller_test.rb	Tests autolocking and provider previews.
test/controllers/content_providers_controller_test.rb	Tests locks, metadata, lookup, and previews.
test/controllers/bioschemas_controller_test.rb	Checks source-free preview persistence.
test/config/test_tess.yml	Enables autolocking in tests.
public/api/definitions/tess.yml	Documents provider ingestion attributes.
lib/scraper.rb	Tracks provider ingestion statistics.
lib/ingestors/provider_ingestion.rb	Resolves, enriches, and writes providers.
lib/ingestors/oai_pmh_ingestor.rb	Uses shared Bioschemas extraction.
lib/ingestors/material_rss_ingestor.rb	Uses shared material extraction.
lib/ingestors/material_ingestion.rb	Preserves provider references.
lib/ingestors/ingestor.rb	Adds source context and provider writes.
lib/ingestors/ingestor_factory.rb	Constructs ingestors from sources.
lib/ingestors/event_rss_ingestor.rb	Uses shared event extraction.
lib/ingestors/event_ingestion.rb	Preserves provider references.
lib/ingestors/bioschemas_ingestor.rb	Centralizes resource extraction.
lib/bioschemas/generator.rb	Exports provider-page links.
lib/bioschemas/event_generator.rb	Exports event providers.
Gemfile.lock	Pins provider-capable extractor revision.
Gemfile	Selects provider-extraction dependency branch.
db/schema.rb	Records provider tracking schema.
db/migrate/20261009120100_add_content_provider_updates_to_sources.rb	Adds provider update counts.
db/migrate/20261009120000_add_scrapable_fields_to_content_providers.rb	Adds provider ingestion history.
db/migrate/20261007141523_add_content_provider_counts_to_sources.rb	Adds provider ingestion counts.
db/migrate/20260826101500_backfill_content_provider_field_locks.rb	Locks existing populated provider fields.
config/tess.example.yml	Documents autolocking configuration.
config/routes.rb	Adds provider preview route.
config/locales/en.yml	Explains portal-only provider ingestion.
config/application.rb	Supports resource-specific autolocking.
app/workers/source_test_worker.rb	Includes providers in test results.
app/views/sources/show.html.erb	Displays provider ingestion counts.
app/views/sources/_test_results.html.erb	Includes provider preview panels.
app/views/sources/_test_resource.html.erb	Displays provider images and fallback notices.
app/views/content_providers/show.html.erb	Supports unsaved provider previews.
app/views/content_providers/partials/_twitter_card.html.erb	Exports keywords and real images.
app/views/content_providers/partials/_content_provider_schemaorg.html.erb	Supports preview metadata rendering.
app/views/content_providers/partials/_content_provider_info.html.erb	Shows preview images and freshness.
app/views/content_providers/new.html.erb	Adds field-lock guidance.
app/views/content_providers/edit.html.erb	Adds field-lock guidance.
app/views/content_providers/_form.html.erb	Adds provider field-lock controls.
app/views/content_providers/_content_provider.html.erb	Displays ingestion freshness.
app/views/common/_image_form.html.erb	Adds image-lock control.
app/views/common/_fairsharing_suggestions.html.erb	Excludes search input from autolocking.
app/views/common/_extra_metadata.html.erb	Handles resources without scientific topics.
app/views/common/_biotools_suggestions.html.erb	Excludes search input from autolocking.
app/views/common/_autocompleter.html.erb	Excludes search input from autolocking.
app/serializers/content_provider_serializer.rb	Exposes provider ingestion attributes.
app/models/source.rb	Passes source context and excludes statistics from activity logs.
app/models/field_lock.rb	Links image locks to image URLs.
app/models/content_provider.rb	Adds locking, freshness, and URL-based identity.
app/models/concerns/has_test_job.rb	Builds and serializes provider previews.
app/helpers/application_helper.rb	Supports preview images and configurable locks.
app/controllers/materials_controller.rb	Uses provider-aware preview construction.
app/controllers/events_controller.rb	Uses provider-aware preview construction.
app/controllers/content_providers_controller.rb	Adds previews and lock enforcement.
app/controllers/bioschemas_controller.rb	Supplies unsaved source context.
app/assets/stylesheets/base/_utilities.scss	Styles provider preview logos.
app/assets/javascripts/field_locks.js	Automatically locks edited fields.
app/assets/javascripts/dropdowns.js	Signals selected-value changes.
app/assets/javascripts/autocompleters.js	Signals selected-value changes.
app/assets/javascripts/ardc_vocabs.js	Signals selected-value changes.

_{Review effort: Balanced}

+      if klass == ContentProvider
+        resource = klass.new(safe_params.merge(extras))


+        fields[:content_provider_id] ||= resource.try(:content_provider)&.id || provider.id
+        fields[:space_id] ||= source.space_id
+        existing_resource = find_existing(type, fields)


+        return candidates.find { |candidate| candidate[:url] == explicit_provider&.url }
+      end
+
+      candidates.find { |candidate| ContentProvider.exists?(url: candidate[:url]) } || candidates.first


eilmiv and others added 15 commits August 25, 2026 13:59

Add keywords to html provider metadata

72d9be2

Add link to further information in TeSS to content provider Bioschemas

e7ef345

Set provider from Bioschemas for "portal" providers

62b5c3b

Implement field locking for content providers

07cd47a

Allow to disable automatic locking of fields in settings

487328f

add provider ingestion module

94bbec3

Implement provider ingestion

4fcf4a4

Refactor provider ingestion

d74c3d4

Small provider ingestion fixes

a42e3c8

Make source a property of an ingestor

53fdbd7

Remove source from filter and write signature of ingestor

b103da8

Implement updating providers

7e7a288

Undo unimportant ingestor changes

524c1a9

Decrease provider status size

c796723

eilmiv requested a balanced review from Copilot October 20, 2026 15:53

Copilot started reviewing on behalf of eilmiv October 20, 2026 15:53

View session

Copilot AI reviewed Oct 20, 2026

View reviewed changes

Navigation Menu

Repository navigation

Exchange content provider - #1388

Exchange content provider#1388
eilmiv wants to merge 15 commits into
ElixirTeSS:masterfrom
pan-training:exchange_content_provider

eilmiv commented Oct 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if klass == ContentProvider
		resource = klass.new(safe_params.merge(extras))

Conversation

eilmiv commented Oct 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants