feat(user-sync): sync OSW User items from MediaWiki and ORCID

docs/tools/user-sync.md

@@ -0,0 +1,87 @@
+# User Item Sync
+
+Create or update OSW `User` items from the MediaWiki accounts of an OSL
+instance, enriching ORCID users from the public ORCID API. The run is
+idempotent and previews every change before writing.
+
+## When to use it
+
+OSL instances gain accounts from manual creation and from whitelisted ORCID
+single sign-on (whose username is the ORCID iD). Those accounts have no
+semantic `User` item until this tool reconciles them.
+
+## Run it
+
+```bash
+uv run python examples/user_sync.py --domain your-instance.example.org --dry-run
+```
+
+Credentials are read from `accounts.pwd.yaml` (or `--cred-filepath`), the same
+mechanism as `OswExpress`. Start with `--dry-run` to preview, then run without
+it to write after confirming at the prompt.
+
+To also remove data no longer wanted (disabled optional fields and
+`employment_contract_status`) from existing items, add `--prune`:
+
+```bash
+uv run python examples/user_sync.py --domain your-instance.example.org --prune --dry-run
+```
+
+Users whose ORCID record exposes no email are listed in a warning at the end of
+the run; a missing email never fails the sync.
+
+### Options
+
+By default the tool syncs both ORCID and MediaWiki-native accounts and writes
+only core identity plus email.
+
+| Flag | Effect |
+| --- | --- |
+| `--domain` | Target OSL domain. |
+| `--cred-filepath` | Path to `accounts.pwd.yaml`. |
+| `--dry-run` | Preview only; never writes. |
+| `--auto-apply` | Non-interactive: apply creates, gap-fills and removals; keep existing on conflicts. |
+| `--limit N` | Process at most N accounts (testing). |
+| `--orcid-only` | Only ORCID-username accounts (excludes `--mw-only`). |
+| `--mw-only` | Only non-ORCID (MediaWiki-native) accounts (excludes `--orcid-only`). |
+| `--include-system` | Do not skip MediaWiki reserved system accounts. |
+| `--no-redirects` | Do not create `User:` redirect pages. |
+| `--with-websites` | Also store ORCID researcher URLs (opt-in). |
+| `--with-organizations` | Also create and link Organization items from ORCID affiliations (opt-in). |
+| `--with-extras` | Enable both websites and organizations. |
+| `--prune` | Remove disabled optional fields and `employment_contract_status` from existing items (default off; a normal run only adds). |
+
+## What it does
+
+1. Enumerate MediaWiki accounts, skipping bots and privileged groups
+   (`bot`, `sysop`, `bureaucrat`, `interface-admin`) and MediaWiki reserved
+   system usernames.
+2. Enrich ORCID users from `https://pub.orcid.org/v3.0/<id>`: names and email
+   always; researcher URLs and affiliations only when their flag is enabled.
+3. Map each account to a proposed `User` item with a deterministic id
+   (uuid5 on the ORCID iD, else the username), so re-runs are idempotent.
+4. Reconcile against existing items by `username` into NEW, GAP_FILL, CONFLICT,
+   REMOVE or UNCHANGED, then preview; gap-fills apply automatically and conflicts
+   are resolved interactively. Removals only appear with `--prune`; a normal run
+   never removes anything.
+5. Store items, create `User:<username>` redirects to each item, ensure any
+   linked Organization items, verify by reloading, and warn about users with no
+   email.
+
+## Data written
+
+| Field | Policy |
+| --- | --- |
+| `username`, `first_name`, `surname`, `label`, `orcid` | Always (core identity). |
+| `email` | Standard: always attempted; missing email is warned, not fatal; never removed. |
+| `website` | Opt-in (`--with-websites`); removed from existing only with `--prune`. |
+| `organization` | Opt-in (`--with-organizations`); removed from existing only with `--prune`. |
+| `employment_contract_status` | Never written; removed from existing only with `--prune`. |
+
+## Notes
+
+- MediaWiki does not expose other users' email or real name, so ORCID is the
+  only source of rich data. Non-ORCID accounts get username-based placeholder
+  names, flagged in the preview.
+- On a first accepted run the library autofetches schemas and may regenerate
+  `src/osw/model/entity.py`; restore it if you do not want that change.

examples/user_sync.py

@@ -0,0 +1,28 @@
+"""Create or update OSW User items from MediaWiki accounts and ORCID.
+
+Usage:
+    uv run python examples/user_sync.py --domain llm4eln.semos.dev --dry-run
+"""
+
+import dotenv
+
+from osw.express import OswExpress
+from osw.tools.user_sync import config_from_args, run_user_sync
+
+
+def main() -> None:
+    dotenv.load_dotenv()
+    config = config_from_args()
+    osw = OswExpress(domain=config.domain, cred_filepath=config.cred_filepath)
+    report = run_user_sync(config, osw=osw)
+    print(report.summary())
+    for title in report.created:
+        print(f"  created: {title}")
+    for title in report.updated:
+        print(f"  updated: {title}")
+    for key, error in report.failed.items():
+        print(f"  FAILED {key}: {error}")
+
+
+if __name__ == "__main__":
+    main()

src/osw/tools/__init__.py

@@ -0,0 +1 @@
+"""Maintenance and automation tools built on top of the osw core library."""

src/osw/tools/user_sync/__init__.py

@@ -0,0 +1,22 @@
+"""Sync OSW User items from MediaWiki accounts and public ORCID data."""
+
+from __future__ import annotations
+
+from .config import (
+    ORGANIZATION_CATEGORY,
+    USER_CATEGORY,
+    SyncConfig,
+    build_arg_parser,
+    config_from_args,
+)
+from .sync import SyncReport, run_user_sync
+
+__all__ = [
+    "ORGANIZATION_CATEGORY",
+    "USER_CATEGORY",
+    "SyncConfig",
+    "SyncReport",
+    "build_arg_parser",
+    "config_from_args",
+    "run_user_sync",
+]

src/osw/tools/user_sync/build.py

@@ -0,0 +1,89 @@
+"""Build pydantic User/Organization entities from proposals.
+
+Must run within an open OSW connection (the models resolve linked entities on
+construction). Organization references are passed as a list of page-title IRIs,
+which oold stores in the entity's ``__iris__`` map; the reference attribute is
+never read back here, as that would trigger a network resolve.
+"""
+
+from __future__ import annotations
+
+from typing import Any, List, Set
+
+from opensemantic.base.v1 import Organization, User
+
+from .config import PROTECTED_FIELDS
+from .mapping import ProposedOrganization, ProposedUser
+
+# The concrete DisplayName type used by the label field.
+_DISPLAY_NAME = User.__fields__["label"].type_
+
+
+def _labels(text: str) -> List[Any]:
+    return [_DISPLAY_NAME(text=text)]
+
+
+def _strip_protected(entity: Any) -> None:
+    """Clear protected relations (e.g. employment_contract_status).
+
+    Set to an empty list rather than removed: the model re-applies its default
+    when the field is absent, and the store only overwrites a relation that is
+    present as an explicit empty value (with ``remove_empty=False``).
+    """
+    iris = getattr(entity, "__iris__", None)
+    if isinstance(iris, dict):
+        for name in PROTECTED_FIELDS:
+            if name in iris:
+                iris[name] = []
+
+
+def build_organization(proposed: ProposedOrganization) -> Organization:
+    return Organization(uuid=proposed.uuid, label=_labels(proposed.name))
+
+
+def build_user(proposed: ProposedUser) -> User:
+    """Build a full User entity for creation (never carries protected fields)."""
+    args: dict = {
+        "uuid": proposed.uuid,
+        "username": proposed.username,
+        "first_name": proposed.first_name,
+        "surname": proposed.surname,
+        "label": _labels(proposed.label),
+    }
+    if proposed.orcid:
+        args["orcid"] = proposed.orcid
+    if proposed.emails:
+        args["email"] = set(proposed.emails)
+    if proposed.websites:
+        args["website"] = set(proposed.websites)
+    if proposed.organizations:
+        args["organization"] = list(proposed.organizations)
+    entity = User(**args)
+    _strip_protected(entity)
+    return entity
+
+
+def apply_update(entity: Any, proposed: ProposedUser, apply_fields: Set[str]) -> Any:
+    """Apply the accepted fields onto a loaded existing entity, in place.
+
+    Fields whose proposal is empty (REMOVE) are cleared; protected fields in
+    ``apply_fields`` are stripped from the entity's ``__iris__`` map.
+    """
+    if "label" in apply_fields:
+        entity.label = _labels(proposed.label)
+    if "first_name" in apply_fields:
+        entity.first_name = proposed.first_name
+    if "surname" in apply_fields:
+        entity.surname = proposed.surname
+    if "orcid" in apply_fields:
+        entity.orcid = proposed.orcid
+    if "emails" in apply_fields:
+        entity.email = set(proposed.emails)
+    if "websites" in apply_fields:
+        entity.website = set(proposed.websites)
+    if "organizations" in apply_fields:
+        # Empty list both links (non-empty) and clears (empty, on removal).
+        entity.__iris__["organization"] = list(proposed.organizations)
+    if any(name in apply_fields for name in PROTECTED_FIELDS):
+        _strip_protected(entity)
+    return entity

src/osw/tools/user_sync/config.py

@@ -0,0 +1,155 @@
+"""Configuration for the user-item sync tool."""
+
+from __future__ import annotations
+
+import argparse
+from dataclasses import dataclass, field
+from typing import List, Optional, Set
+
+# Category page titles of the target item types (see opensemantic.base.v1).
+USER_CATEGORY = "Category:OSWd9aa0bca9b0040d8af6f5c091bf9eec7"
+ORGANIZATION_CATEGORY = "Category:OSW1969007d5acf40539642877659a02c23"
+
+# Public ORCID API used to enrich ORCID users.
+ORCID_API_BASE_DEFAULT = "https://pub.orcid.org/v3.0"
+
+# MediaWiki groups whose members are treated as bot/system accounts and skipped.
+# All are standard MediaWiki groups, so this is not a per-instance skip-list.
+SYSTEM_GROUPS = ["bot", "sysop", "bureaucrat", "interface-admin"]
+
+# User fields the script must never write and must remove from existing items
+# (data protection).
+PROTECTED_FIELDS = ("employment_contract_status",)
+
+
+@dataclass
+class SyncConfig:
+    """Runtime options for a single sync run."""
+
+    domain: Optional[str] = None
+    cred_filepath: Optional[str] = None
+    dry_run: bool = False
+    auto_apply: bool = False
+    limit: Optional[int] = None
+    include_orcid: bool = True
+    include_non_orcid: bool = True
+    exclude_bot_group: bool = True
+    exclude_system_usernames: bool = True
+    create_redirects: bool = True
+    # Optional enrichment beyond core identity, opt-in (off by default).
+    # Email is not optional: it is always attempted, and missing emails are
+    # reported as warnings rather than failing the run.
+    include_websites: bool = False
+    link_organizations: bool = False
+    # Remove disabled optional fields and protected fields from existing items.
+    prune: bool = False
+    orcid_api_base: str = ORCID_API_BASE_DEFAULT
+    excluded_groups: List[str] = field(default_factory=lambda: list(SYSTEM_GROUPS))
+
+    def enabled_optional_fields(self) -> Set[str]:
+        """Reconcile field names for the enabled optional data."""
+        enabled: Set[str] = set()
+        if self.include_websites:
+            enabled.add("websites")
+        if self.link_organizations:
+            enabled.add("organizations")
+        return enabled
+
+
+def build_arg_parser() -> argparse.ArgumentParser:
+    """Build the argument parser used by the example wrapper."""
+    parser = argparse.ArgumentParser(
+        description="Create or update OSW User items from MediaWiki and ORCID.",
+    )
+    parser.add_argument("--domain", help="Target OSL domain, e.g. llm4eln.semos.dev.")
+    parser.add_argument(
+        "--cred-filepath",
+        dest="cred_filepath",
+        help="Path to accounts.pwd.yaml (defaults to the working directory).",
+    )
+    parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        help="Show the preview only; do not write anything.",
+    )
+    parser.add_argument(
+        "--auto-apply",
+        dest="auto_apply",
+        action="store_true",
+        help="Non-interactive: apply creates, gap-fills and removals; keep existing on conflicts.",
+    )
+    parser.add_argument(
+        "--limit",
+        type=int,
+        help="Process at most this many MediaWiki users (for testing).",
+    )
+    parser.add_argument(
+        "--no-redirects",
+        dest="create_redirects",
+        action="store_false",
+        help="Do not create User: redirect pages.",
+    )
+    parser.add_argument(
+        "--with-websites",
+        dest="include_websites",
+        action="store_true",
+        help="Also store ORCID researcher URLs on the user item (opt-in).",
+    )
+    parser.add_argument(
+        "--with-organizations",
+        dest="link_organizations",
+        action="store_true",
+        help="Also resolve ORCID affiliations to Organization items and link them.",
+    )
+    parser.add_argument(
+        "--with-extras",
+        dest="with_extras",
+        action="store_true",
+        help="Enable all optional data: websites and organizations.",
+    )
+    parser.add_argument(
+        "--prune",
+        dest="prune",
+        action="store_true",
+        help="Remove disabled optional fields and employment_contract_status from "
+        "existing items (data cleanup). Off by default; a normal run only adds.",
+    )
+    account_scope = parser.add_mutually_exclusive_group()
+    account_scope.add_argument(
+        "--orcid-only",
+        dest="include_non_orcid",
+        action="store_false",
+        help="Sync only accounts whose username is an ORCID iD.",
+    )
+    account_scope.add_argument(
+        "--mw-only",
+        dest="include_orcid",
+        action="store_false",
+        help="Sync only non-ORCID (MediaWiki-native) accounts.",
+    )
+    parser.add_argument(
+        "--include-system",
+        dest="exclude_system_usernames",
+        action="store_false",
+        help="Do not skip MediaWiki reserved system accounts (Maintenance script etc.).",
+    )
+    return parser
+
+
+def config_from_args(argv: Optional[List[str]] = None) -> SyncConfig:
+    """Parse command-line arguments into a SyncConfig."""
+    args = build_arg_parser().parse_args(argv)
+    return SyncConfig(
+        domain=args.domain,
+        cred_filepath=args.cred_filepath,
+        dry_run=args.dry_run,
+        auto_apply=args.auto_apply,
+        limit=args.limit,
+        include_orcid=args.include_orcid,
+        include_non_orcid=args.include_non_orcid,
+        exclude_system_usernames=args.exclude_system_usernames,
+        create_redirects=args.create_redirects,
+        include_websites=args.include_websites or args.with_extras,
+        link_organizations=args.link_organizations or args.with_extras,
+        prune=args.prune,
+    )