[sync] Synced updates by the same rule as deletions, and removed syncDeletions.

.github/workflows/sync.yml

@@ -54,7 +54,7 @@ jobs:
           repository: SgtSwagrid/github-graph
           path: .github-graph
 
-      # Full history of both repositories is needed to sync deletions.
+      # Full history of both repositories is needed to tell the source's changes from the target's.
       - name: Checkout source
         uses: actions/checkout@v4
         with:

README.md

@@ -82,6 +82,29 @@ Changes _pushed_ to the source are _eagerly_ propagated (i.e. _pushed_) downstre
 A [GitHub Actions](https://github.com/features/actions) workflow in the source repository listens for pushes to a designated branch and directory,
 in response to which pull requests are automatically opened.
 
+### Which files are synced
+
+Every file which the source has ever had (outside the [`ignore`](#ignore) list) is given the state it now has in the source:
+its latest version, or none if the source has deleted it.
+But a target may change a synced file itself, or have a file of its own with the same name,
+so the source's state is only given to a file whose last change in the target came from the source.
+That is, the history of both repositories shows that:
+- **its state in the target is one which the source had before the target last changed it**,
+  where a file's absence counts as one of the source's states once the source has deleted it, and
+- **the source has changed it since**.
+
+A file which the target never had is simply added.
+So a file which the target has modified, deleted, or re-added after its deletion was synced is left alone from then on,
+as is one which the target had of its own before the source added it.
+Each is reported in the workflow logs as `Kept`.
+To have a file follow the source again, give it the source's latest version (or delete it, if the source has deleted it),
+and the source's next change to it is synced as usual.
+
+Only a file's last change counts, not how it first arrived:
+a file which the source overwrote, or which another source first added before this one took it over (as when a repository is moved beneath a new parent),
+is the source's as long as its version is one which the source had.
+A renamed file is treated as a deletion of the old name, plus an addition of the new one.
+
 ### One pull request per target
 
 Each branch of a target repository has a single sync branch, and so at most one open pull request,
@@ -93,7 +116,7 @@ The sync branch holds one commit for each source directory with changes, on top
 Each sync rebuilds it on the latest target branch,
 reapplying the commits from every other source directory, and replacing its own with its latest changes
 (or dropping it, if there are none).
-As with [`syncDeletions`](#syncdeletions), a deletion isn't reapplied if the file has since been modified in the target branch.
+As in [which files are synced](#which-files-are-synced), a change isn't reapplied to a file which the target branch has changed itself since the change was made.
 Anything else pushed to the sync branch is discarded.
 If several syncs update the branch at once, only one can push at a time,
 and the others rebuild the branch with its changes and try again.
@@ -195,33 +218,7 @@ Generally, you'll at least want to ignore the _GitHub Graph_ setup itself,
 i.e. `.github/workflows/sync.yml` and `.github/graph.json`,
 as these aren't excluded automatically.
 It is not necessary to ignore files which lie outside of `source.root`.
-
-### `syncDeletions`
-
-Whether files which are deleted from the source should also be deleted from the target.
-Can be defined for a child, or globally at the top-level.
-Defaults to `true`.
-
-```json
-{
-  "syncDeletions": false
-}
-```
-
-As the target may have files of its own which happen to share a name with one deleted from the source,
-a file is only deleted if the history of both repositories shows that its last change in the target came from the source:
-that it's still a version which the source once had, and was last changed before the source deleted it.
-In particular, a file is kept if:
-- **It was modified in the target.** That is, it doesn't match any version which the source ever had.
-- **It was changed or re-added after the deletion.** For example, the deletion was synced, but the target later restored the file or created a new one with the same name.
-
-Only the file's last change counts, not how it first arrived.
-So a file which the target had first, but which the source then overwrote, is the source's from then on.
-Likewise for one which another source first added, before the current one took it over, as when a repository is moved beneath a new parent.
-
-Any file which is kept for one of these reasons is reported in the workflow logs.
-A renamed file is treated as a deletion of the old name, plus an addition of the new one.
-Files in the `ignore` list are never deleted.
+Files in the `ignore` list are never added, updated or deleted.
 
 ### `token`
 
@@ -309,11 +306,12 @@ That being said, if you wanted to tackle these yourself, I'd be a grateful PR re
 
 ### Merge semantics
 
-Updated files are never "merged", but simply overwrite whatever exists downstream.
-Likewise, deleted files are deleted downstream, unless they've since been modified or re-added there (see [`syncDeletions`](#syncdeletions)).
-_GitHub Graph_ is only intended for use when the responsibility for each file can be unambiguously associated with a single source repository,
-with the understanding that copies shouldn't be modified.
-If several sources nonetheless sync the same file into one target, whichever synced most recently wins.
+Files are never "merged": each is either the source's, and follows the source, or the target's, and is left alone
+(see [which files are synced](#which-files-are-synced)).
+A file changed downstream stops following the source altogether, however small the change.
+_GitHub Graph_ is only intended for use when the responsibility for each file can be unambiguously associated with a single source repository.
+If several sources nonetheless sync the same file into one target, the first to sync it keeps it,
+unless another has had the very same version.
 
 ### Platform support
 

scripts/load-config.sh

@@ -30,7 +30,6 @@ jq -r '
   "TARGET_ROOT=\(.target.root)",
   "TARGET_URL=\(.target.url)",
   "IGNORE=\(.ignore | tojson)",
-  "SYNC_DELETIONS=\(.syncDeletions)",
   "PR_TITLE=\(.pullRequest.title)"
 ' <<< "$CHILD" >> "$GITHUB_ENV"
 

scripts/merge-files.sh

@@ -2,13 +2,23 @@
 set -euo pipefail
 
 # =================================================================================================
-# Locally applies updates from the source repository into the target repository.
+# Locally applies the changes from the source repository to the target repository.
+#
+# Every file which the source has ever had in SOURCE_ROOT is given the state it now has in the
+# source: its current version, or none if it was deleted. But the target may change a file itself,
+# or have a file of its own with the same name, so the source's state is only given to a file
+# whose last change in the target came from the source:
+#   a) Its state in the target is one which the source had before the target last changed it.
+#      A file's absence counts as a state which the source had once the source deleted it.
+#   b) The source has changed it since the target last changed it.
+# A file which the target never had is given the source's state.
+# So a file which the target has modified, deleted or re-added is left alone from then on,
+# as is one which the target had of its own before the source added it.
 #
 # Environment variables:
 #   - IGNORE: A JSON-formatted list of file paths to exclude from syncing.
 #   - SOURCE_ROOT: The root directory within the source repository to copy files from.
 #   - TARGET_ROOT: The root directory within the target repository to copy files into.
-#   - SYNC_DELETIONS: Whether to delete files from the target which were deleted from the source.
 #
 # Requirements:
 #   - source/ should already contain the source repository at the correct branch, with full history.
@@ -52,6 +62,7 @@ target_relative_path() {
 # Output the first-parent history of a branch, as NUL-separated records of the form:
 #   <commit time> <blob> <status> <file>
 # There is one record for each file changed by each commit, with renames treated as a deletion plus an addition.
+# The blob of a deleted file is ABSENT.
 # Following only the first parent means that each change is dated by when it actually landed on the branch.
 #   - $1: The repository directory.
 #   - $@: Any further arguments are passed to `git log`.
@@ -71,92 +82,67 @@ file_history() {
   )
 }
 
-# =================================================================================================
-# 1. Copy all files in the SOURCE_ROOT directory of the source repository
-#    to the TARGET_ROOT directory of the target repository.
-#    Files in the ignore list or outside of SOURCE_ROOT are skipped.
-#    Files which already exist in the target repository are replaced.
-# =================================================================================================
-
-# A set of names of all files which lie in the SOURCE_ROOT directory of the source repository.
-declare -A SOURCE_FILES
-while IFS= read -r line; do
-  SOURCE_FILES["$line"]=1;
-done < <(git -C source ls-files -- "$SOURCE_ROOT")
-
-# Copy every tracked file from the source to the target repository.
-for file in "${!SOURCE_FILES[@]}"; do
-  relative="${file#"$SOURCE_ROOT/"}"
-  if is_ignored "$relative"; then
-    echo "Skipped: $file"
-  else
-    dest=$(target_path "$file")
-    mkdir -p "$(dirname "$dest")"
-    cp "source/$file" "$dest"
-    echo "Copied: $file"
-  fi
-done
-
-# =================================================================================================
-# 2. Delete from the target repository all files which were synced from the source repository,
-#    but which have since been deleted from the source repository.
-#    Files in the ignore list or outside of SOURCE_ROOT are skipped.
-#    As the target may have files of its own with the same names, a file is only deleted if
-#    its last change in the target came from the source, before the source deleted it:
-#      a) It hasn't been modified in the target.
-#         i.e. It currently matches a version which the source once had.
-#      b) It hasn't been changed or re-added in the target since being deleted from the source.
-#         i.e. It was last changed in the target before it was deleted from the source.
-#    So a file which the target had first, but which the source then overwrote, is the source's
-#    from then on, as is one which an earlier source first added: only its last change counts.
-# =================================================================================================
-
-if [[ "$SYNC_DELETIONS" != "true" ]]; then
-  echo "Skipped deletions, as syncDeletions is disabled."
-  exit 0
-fi
+# The state of a file which doesn't exist: the blob which git gives a deleted file, all zeros.
+ABSENT=$(git -C source hash-object --stdin < /dev/null | tr '0-9a-f' '0')
 
-# Provenance can't be determined without the full history of both repositories.
+# Whose change was whose can't be told without the full history of both repositories.
 if [[ "$(git -C source rev-parse --is-shallow-repository)" == "true" ||
       "$(git -C target rev-parse --is-shallow-repository)" == "true" ]]; then
-  echo "::warning::Skipped deletions, as they require the full history of both repositories."
-  exit 0
+  echo "::error::Syncing requires the full history of both repositories."
+  exit 1
 fi
 
-# For every file in SOURCE_ROOT which was ever in the source repository:
-#   - DELETED_AT: The time at which it was deleted from the source, or empty if it still exists.
-#   - SOURCE_VERSIONS: The set of versions it has had in the source, keyed by "<blob>:<file>".
-declare -A DELETED_AT SOURCE_VERSIONS
+# =================================================================================================
+# 1. Read the history of every file which the source has ever had in SOURCE_ROOT:
+#      - STATE: Its state now, i.e. its current blob, or ABSENT if it has been deleted.
+#      - CHANGED: When the source last changed it.
+#      - SINCE: When the source first had each of its states, keyed by "<state>:<file>".
+# =================================================================================================
+
+declare -A STATE CHANGED SINCE
 while IFS= read -r -d '' timestamp && IFS= read -r -d '' blob && IFS= read -r -d '' status && IFS= read -r -d '' file; do
-  if [[ "$status" == "D" ]]; then
-    DELETED_AT["$file"]="$timestamp"
-  else
-    DELETED_AT["$file"]=""
-    SOURCE_VERSIONS["$blob:$file"]=1
-  fi
+  STATE["$file"]="$blob"
+  CHANGED["$file"]="$timestamp"
+  if [[ -z "${SINCE["$blob:$file"]+_}" ]]; then SINCE["$blob:$file"]="$timestamp"; fi
 done < <(file_history source --reverse -- "$SOURCE_ROOT")
 
-for file in "${!DELETED_AT[@]}"; do
-  deleted_at="${DELETED_AT[$file]}"
+# =================================================================================================
+# 2. Give each file in the target the state it has in the source, if the file is the source's.
+#    Files in the ignore list are skipped.
+# =================================================================================================
 
-  # Ignore because file still exists in the source.
-  if [[ -z "$deleted_at" ]]; then continue; fi
-  # Ignore because file is in ignore list.
-  if is_ignored "${file#"$SOURCE_ROOT/"}"; then continue; fi
+for file in "${!STATE[@]}"; do
+  if is_ignored "${file#"$SOURCE_ROOT/"}"; then
+    echo "Skipped: $file"
+    continue
+  fi
 
-  # Ignore because file doesn't exist in the target.
+  wanted="${STATE[$file]}"
   dest=$(target_relative_path "$file")
-  current=$(git -C target rev-parse -q --verify "HEAD:$dest") || continue
+  current=$(git -C target rev-parse -q --verify "HEAD:$dest") || current="$ABSENT"
+  if [[ "$current" == "$wanted" ]]; then continue; fi
 
-  # When the file was last changed in the target, dated by when the change landed on the branch.
+  # When the target last changed the file, dated by when the change landed on its branch,
+  # or nothing if it never had the file.
   changed_at=$(git -C target log -1 --first-parent --format=%ct -- ":(literal)$dest")
+  if [[ -n "$changed_at" ]]; then
+    since="${SINCE["$current:$file"]:-}"
+    source_changed_at="${CHANGED[$file]}"
+    if [[ -z "$since" ]] || (( since > changed_at )); then
+      echo "Kept: $file (changed in the target)"
+      continue
+    elif (( source_changed_at <= changed_at )); then
+      echo "Kept: $file (changed in the target since the source last changed it)"
+      continue
+    fi
+  fi
 
-  if [[ -z "${SOURCE_VERSIONS["$current:$file"]+_}" ]]; then
-    echo "Kept: $file (doesn't match any version which the source had)"
-  elif [[ -z "$changed_at" ]] || (( changed_at >= deleted_at )); then
-    echo "Kept: $file (changed or re-added after being deleted from the source)"
-  else
+  if [[ "$wanted" == "$ABSENT" ]]; then
     rm "target/$dest"
     echo "Deleted: $file"
+  else
+    mkdir -p "$(dirname "target/$dest")"
+    cp "source/$file" "target/$dest"
+    if [[ "$current" == "$ABSENT" ]]; then echo "Added: $file"; else echo "Updated: $file"; fi
   fi
 done

scripts/parse-config.sh

@@ -72,10 +72,6 @@ CHILDREN=$(jq \
     map(
       .token //= "GH_TOKEN"
     ) |
-    # Note that `//=` would also replace an explicit `false`.
-    map(
-      .syncDeletions |= (if . == null then true else . end)
-    ) |
 
     # Filter to only children that are triggered by the current branch.
     map(select(.source.branch == $ENV.GITHUB_REF_NAME))

scripts/publish.sh

@@ -119,28 +119,29 @@ key_of() {
   git log -1 --format="%(trailers:key=$TRAILER,valueonly)" "$1" | sed -n 1p
 }
 
+# The state of a file which doesn't exist: the blob which git gives a deleted file, all zeros.
+ABSENT=$(git hash-object --stdin < /dev/null | tr '0-9a-f' '0')
+
 # Apply the changes made by a commit on top of another, as whole files, and output the new commit,
 # or nothing if there are no changes left to make.
-# As in `merge-files.sh`, a deletion is skipped if the file has been modified since.
+# As in `merge-files.sh`, a file which the target has changed itself is left alone:
+# each change, whether it adds, updates or deletes a file, is only applied where the file is still
+# as it was when the change was made.
 #   - $1: The commit to apply the changes on top of.
 #   - $2: The commit which made the changes.
 replay() {
-  local onto="$1" commit="$2" index tree meta path new_mode old_blob new_blob status current
+  local onto="$1" commit="$2" index tree meta path new_mode old_blob new_blob current
   index="$(git rev-parse --git-dir)/github-graph-index"
   rm -f "$index"
   GIT_INDEX_FILE="$index" git read-tree "$onto"
   while IFS= read -r -d '' meta && IFS= read -r -d '' path; do
-    read -r _ new_mode old_blob new_blob status <<< "$meta"
-    if [[ "$status" != "D" ]]; then
-      printf '%s %s\t%s\0' "$new_mode" "$new_blob" "$path"
-      continue
-    fi
-    current=$(git rev-parse -q --verify "$onto:$path") || continue
+    read -r _ new_mode old_blob new_blob _ <<< "$meta"
+    current=$(git rev-parse -q --verify "$onto:$path") || current="$ABSENT"
     if [[ "$current" == "$old_blob" ]]; then
-      # Mode 0 removes the file.
-      printf '0 %s\t%s\0' "$old_blob" "$path"
-    else
-      echo "Kept: $path (modified in $TARGET_BRANCH since its deletion was synced)" >&2
+      # A deleted file has mode 0, which removes it.
+      printf '%s %s\t%s\0' "$new_mode" "$new_blob" "$path"
+    elif [[ "$current" != "$new_blob" ]]; then
+      echo "Kept: $path (changed in $TARGET_BRANCH since it was synced)" >&2
     fi
   done < <(git diff-tree -r -z --no-renames "$commit^" "$commit") |
     GIT_INDEX_FILE="$index" git update-index -z --index-info

templates/pull-request-body.md

@@ -9,8 +9,8 @@ Subsequent commits to [$SOURCE_BRANCH]($SOURCE_BRANCH_URL) will also be incorpor
 
 ### Warning
 
-This change is potentially destructive, as files with matching names will be overwritten,
-and files which were deleted upstream will also be deleted here.
+This changes, adds and deletes files here as they were changed, added and deleted upstream,
+leaving alone any file which was changed here since it was last synced.
 Please review carefully.
 
 ### Explanation