From d6f334b861ca30d13001bc4a6b325706b475d81b Mon Sep 17 00:00:00 2001 From: SBPro Date: Tue, 17 Mar 2026 12:48:17 +0300 Subject: [PATCH] Add linearize fallback for reverse sync and workflow guide (v1.3) When josh-proxy rejects a reverse sync push due to unmappable merge commits, fall back to linearizing: cherry-pick regular commits individually, squash only the merge commits via cherry-pick -m 1. Also adds a recommended Git workflow section to the guide explaining where cross-branch merges should happen (monorepo) vs feature work (subrepo), and expands troubleshooting for the "josh rejected push" error with root cause analysis and prevention advice. Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 11 +++ VERSION | 2 +- docs/adr/011-linearize-fallback-reverse.md | 68 +++++++++++++ docs/adr/README.md | 1 + docs/guide.md | 84 +++++++++++++++- lib/sync.sh | 106 +++++++++++++++++---- 6 files changed, 253 insertions(+), 19 deletions(-) create mode 100644 docs/adr/011-linearize-fallback-reverse.md diff --git a/CHANGELOG.md b/CHANGELOG.md index dd3631e..98d27db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## 1.3.0 + +### Features + +- **Linearize fallback for reverse sync**: When josh-proxy rejects a push due to unmappable merge commits, reverse sync automatically falls back to linearizing the history — cherry-picking regular commits individually and squashing only the problematic merge commits via `cherry-pick -m 1`. Preserves individual commit granularity while handling complex merge topologies that josh cannot map. PR body notes when linearization was used. + +### Docs + +- Expanded troubleshooting guide for "Josh rejected push" with root cause analysis and prevention advice. +- Added ADR-011: Linearize fallback for reverse sync. + ## 1.2.0 ### Features diff --git a/VERSION b/VERSION index 26aaba0..f0bb29e 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.2.0 +1.3.0 diff --git a/docs/adr/011-linearize-fallback-reverse.md b/docs/adr/011-linearize-fallback-reverse.md new file mode 100644 index 0000000..80e3261 --- /dev/null +++ b/docs/adr/011-linearize-fallback-reverse.md @@ -0,0 +1,68 @@ +# ADR-011: Linearize Fallback for Reverse Sync + +**Status:** Accepted +**Date:** 2026-03 + +## Context + +Josh-proxy rejects reverse sync pushes when the subrepo history contains merge commits whose parents it cannot map through the filter. This happens when: + +1. A long-lived branch (e.g., `stage`) is merged into `main` via a merge commit +2. That branch contains auto-sync merge commits or criss-cross merges with `main` +3. Josh encounters the merge commit, tries to map both parents through the filter, and fails with a 500 error + +This is a legitimate subrepo workflow — teams merge staging branches into main regularly. The sync tool should handle it without requiring teams to change their Git workflow. + +### Alternatives considered + +1. **Fail and log**: The pre-v1.3 behavior. Leaves the sync stuck until someone manually intervenes. Bad for unattended cron-based sync. + +2. **Squash all commits into one**: Create a single `commit-tree` with the diff between josh-filtered base and subrepo HEAD. Simple but destroys all commit granularity — 10 unrelated commits appear as one blob on the monorepo PR. + +3. **Rewrite subrepo history**: Rebase or filter-branch the subrepo to remove problematic merges. Breaks the sync relationship with the monorepo (josh's SHA mapping becomes invalid) and forces all developers to re-clone. + +4. **Linearize: cherry-pick regular commits, squash only merges**: Walk the human commits in order, cherry-pick non-merge commits as-is, and use `cherry-pick -m 1` for merge commits. Preserves individual commit granularity for regular commits; only the problematic merge commits lose their multi-parent structure. + +## Decision + +When the direct push through josh-proxy fails, fall back to option 4: linearize the history by cherry-picking onto the josh-filtered base. + +### How it works + +1. Direct `git push` through josh-proxy is attempted first (existing behavior) +2. If it fails, create a temporary branch from `mono-filtered/` +3. Walk human commits (oldest-first, bot commits excluded) from the ancestry path: + - **Regular commits** (≤1 parent): `git cherry-pick ` — preserves author, date, and message + - **Merge commits** (>1 parent): `git cherry-pick -m 1 ` — applies the merge's diff relative to its first parent as a single commit +4. If cherry-pick conflicts (rare — usually due to ordering issues), fall back to `git diff | git apply` with the original author metadata +5. Push the linearized branch through josh-proxy +6. PR body includes a note explaining that merge commits were squashed + +### What is preserved + +- Individual non-merge commits (author, date, message, diff) +- The net effect of each merge commit (as a squashed single commit) +- The original commit list in the PR body for reference + +### What is lost + +- The multi-parent structure of merge commits (they become single-parent) +- The distinction between "changes introduced by the merge" vs "changes from each parent" — the merge is represented as its diff from first parent + +## Consequences + +**Positive:** +- Reverse sync no longer gets stuck on merge commits — handles the common case automatically +- Non-merge commits retain full granularity (no unnecessary squashing) +- Subrepo history is untouched — no rewriting, no broken sync relationship +- The PR body documents when linearization was used, so reviewers know + +**Negative:** +- Merge commit semantics are lost on the monorepo side (they appear as regular commits) +- If a merge commit's changes conflict with a prior cherry-picked commit during linearization, the diff-apply fallback may produce a subtly different result than the original merge resolution +- Adds complexity to the reverse sync path (two code paths: direct push and linearize fallback) + +**Risk mitigation:** +- The direct push is always tried first — linearization only activates when josh rejects the push +- The monorepo PR is still reviewed by humans before merging +- The original commit SHAs are listed in the PR body for cross-referencing diff --git a/docs/adr/README.md b/docs/adr/README.md index 42e2773..eefee96 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -16,3 +16,4 @@ This directory contains Architecture Decision Records (ADRs) for josh-sync. Each | [008](008-first-parent-ordering.md) | First-parent ordering in reconciliation merges | Accepted | | [009](009-tree-comparison-guard.md) | Tree comparison as sync skip guard | Accepted | | [010](010-onboard-checkpoint-resume.md) | Onboard workflow with checkpoint/resume | Accepted | +| [011](011-linearize-fallback-reverse.md) | Linearize fallback for reverse sync | Accepted | diff --git a/docs/guide.md b/docs/guide.md index bb1cc3e..38f4a67 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -518,6 +518,60 @@ Bot commits include a git trailer like `Josh-Sync-Origin: forward/main/2024-02-1 Sync state is stored as JSON files on an orphan branch (`josh-sync-state`), one file per target/branch. This tracks the last-synced commit SHAs and timestamps to avoid re-syncing the same changes. +## Recommended Git Workflow + +Josh-proxy maps commits through a filter per-branch. It handles linear history and simple merges (short-lived feature branches) without issues. However, it **cannot map merge commits whose parents were created on the subrepo side** — because those commits were never pushed through josh and have no monorepo-side mapping. + +This means cross-branch merges (e.g., `stage` → `main`) must happen on the monorepo side, where josh can filter the result cleanly. + +### The rule + +**The monorepo owns branch topology. The subrepo owns feature development.** + +### What to do where + +| Action | Where to do it | Why | +|--------|---------------|-----| +| Feature branch → `main` | **Subrepo** (PR, any merge strategy) | Short-lived branch with clean lineage — josh handles it | +| `stage` → `main` (promotion) | **Monorepo** | Cross-branch merge — forward sync propagates the result to both subrepo branches | +| `main` → `stage` (catch-up) | **Monorepo** | Same reason — avoids criss-cross merge history on subrepo | +| Hotfix to `main` | **Either side** | Single commit or small PR — works everywhere | +| Config/CI changes (monorepo-only) | **Monorepo** | Not synced to subrepo (use `exclude` for monorepo-only files) | + +### Feature development (subrepo) + +This is the primary workflow for subrepo developers: + +1. Create a feature branch from `main` (or whichever synced branch) +2. Develop, commit, push +3. Open a PR targeting `main` on the subrepo +4. Merge the PR (merge commit, squash, or rebase — all work) +5. Reverse sync picks up the new commits and creates a PR on the monorepo + +Any merge strategy works because the feature branch lineage stays within josh's mapped history. + +### Cross-branch merges (monorepo) + +When promoting `stage` to `main`, or catching up `stage` with `main`: + +1. Open a PR on the **monorepo** merging `stage` → `main` (or `main` → `stage`) +2. Review and merge on the monorepo +3. Forward sync propagates the result to the subrepo's `main` and `stage` branches + +This works because josh does the filtering — it computes the subrepo view from the monorepo merge result, rather than trying to reconstruct a monorepo merge from subrepo commits. + +### What to avoid + +- **Don't merge `stage` into `main` on the subrepo with a merge commit.** The merge parents include commits created on the subrepo side (forward sync merges, criss-cross merges) that josh has no mapping for. Josh rejects the push with a 500 error. +- **Don't merge `main` into `stage` on the subrepo.** Creates criss-cross merge history that causes the same josh mapping failure when `stage` is later merged back. +- **Don't rebase synced branches on the subrepo.** This rewrites commit SHAs that josh has already mapped, breaking the sync relationship. + +### If you must merge cross-branch on the subrepo + +Use a **squash merge**. A squash merge produces a single commit with one parent — josh can always map it. You lose the individual commit history on the target branch, but the sync goes through cleanly. + +As a safety net, josh-sync v1.3+ automatically falls back to linearizing the history when josh rejects a push — cherry-picking regular commits individually and squashing only the problematic merge commits. See the [troubleshooting section](#josh-rejected-push-reverse-sync) and [ADR-011](adr/011-linearize-fallback-reverse.md). + ## Excluding Files from Sync Some files in the monorepo subfolder may not belong in the subrepo (e.g., monorepo-specific CI configs, internal tooling). The `exclude` config field removes these at the josh-proxy layer — excluded files never appear in the subrepo. @@ -616,7 +670,35 @@ Normal: the subrepo changed while sync was running. The next sync run will pick ### "Josh rejected push" (reverse sync) -Josh-proxy couldn't map the push back to the monorepo. Check josh-proxy logs, verify the josh filter is correct. May indicate a history divergence — consider running `josh-sync reset `. +Josh-proxy couldn't map the push back to the monorepo. This has two common causes: + +#### Merge commits with unmappable parents + +**Symptom:** Josh returns `500 Internal Server Error` with a message like: + +``` +rejecting merge with 2 parents: +"Merge pull request 'stage' (#30) from stage into main" (c4fa3c9...) +1) "Merge branch 'auto-sync/import-...'" (4bf8704...) +2) "Merge branch 'main' into stage" (d021654...) +``` + +**Cause:** The subrepo has a merge commit whose parents josh-proxy cannot trace through its filter. This typically happens when: +- A long-lived branch (e.g., `stage`) is merged into `main` via a merge commit (not squash) +- That branch contains auto-sync merge commits or other history that doesn't exist in josh's filtered view +- Someone merges `main` into a feature/staging branch and then merges it back — the criss-cross parents confuse josh's mapping + +**Automatic handling (v1.3+):** josh-sync automatically falls back to linearizing the history when the direct push fails. Regular commits are cherry-picked individually (preserving authorship and messages), while merge commits are squashed into single commits via `cherry-pick -m 1`. The PR notes when this fallback was used. See [ADR-011](adr/011-linearize-fallback-reverse.md). + +**Prevention:** In josh-synced subrepos, prefer **squash merges** when merging long-lived branches (stage, develop) into the synced branch. Squash merges produce a single commit with no merge parents, which josh can always map. Regular feature branch merges (short-lived, no auto-sync history) are usually fine. + +**Manual resolution (if automatic fallback also fails):** This indicates a more fundamental history issue. Options: +1. Cherry-pick the desired changes manually onto a clean branch and push through josh +2. Run `josh-sync reset ` to re-establish history (destructive — all subrepo clones must re-fetch) + +#### Filter or path mismatch + +Josh-proxy couldn't map the push due to an incorrect filter. Check josh-proxy logs, verify the `josh_filter` or `subfolder` in `.josh-sync.yml` is correct, and ensure the subfolder exists in the monorepo. ### Import PR shows "No changes" diff --git a/lib/sync.sh b/lib/sync.sh index 22b9888..95c2e51 100644 --- a/lib/sync.sh +++ b/lib/sync.sh @@ -284,37 +284,109 @@ reverse_sync() { local ts ts=$(date +%Y%m%d-%H%M%S) local staging_branch="auto-sync/subrepo-${subrepo_branch}-${ts}" + local push_ref="HEAD" + local linearized=false - if git push -o "base=${mono_branch}" "$(josh_auth_url)" "HEAD:refs/heads/${staging_branch}"; then - log "INFO" "Pushed to staging branch via josh: ${staging_branch}" + if ! git push -o "base=${mono_branch}" "$(josh_auth_url)" "${push_ref}:refs/heads/${staging_branch}" 2>/dev/null; then + # Josh rejects pushes when the history contains merge commits whose + # parents it cannot map (e.g., merges of staging/feature branches that + # include auto-sync history). Fall back to linearizing: cherry-pick + # regular commits as-is, squash only the problematic merge commits. + # See ADR-011. + log "WARN" "Direct push failed — linearizing history (cherry-pick + squash merges)" - # 6. Create PR on monorepo (NEVER direct push) - local pr_body - pr_body="## Subrepo changes + local original_head + original_head=$(git rev-parse HEAD) + + git checkout -b "tmp-linearize-$$" "mono-filtered/${mono_branch}" 2>/dev/null \ + || die "Failed to create linearize branch" + + local commit_shas + commit_shas=$(git log --ancestry-path "mono-filtered/${mono_branch}..${original_head}" \ + --reverse --format="%H" --invert-grep --grep="^${BOT_TRAILER}:" 2>/dev/null || echo "") + + local sha + for sha in $commit_shas; do + local parent_count + parent_count=$(git cat-file -p "$sha" | grep -c "^parent ") + + if [ "$parent_count" -le 1 ]; then + # Regular commit — cherry-pick to preserve message and authorship + if ! git cherry-pick "$sha" 2>/dev/null; then + log "WARN" "Cherry-pick conflict on ${sha:0:7} — applying tree directly" + git cherry-pick --abort 2>/dev/null + # Fall back to applying the exact tree diff for this commit + local msg author_name author_email author_date + msg=$(git log -1 --format="%B" "$sha") + author_name=$(git log -1 --format="%an" "$sha") + author_email=$(git log -1 --format="%ae" "$sha") + author_date=$(git log -1 --format="%aI" "$sha") + git diff HEAD "$sha" | git apply --index 2>/dev/null \ + || die "Failed to apply diff for ${sha:0:7}" + GIT_AUTHOR_NAME="$author_name" GIT_AUTHOR_EMAIL="$author_email" \ + GIT_AUTHOR_DATE="$author_date" \ + git commit -m "$msg" || die "Failed to commit ${sha:0:7}" + fi + else + # Merge commit — cherry-pick relative to first parent (squashes the merge) + local msg author_name author_email author_date + msg=$(git log -1 --format="%B" "$sha") + author_name=$(git log -1 --format="%an" "$sha") + author_email=$(git log -1 --format="%ae" "$sha") + author_date=$(git log -1 --format="%aI" "$sha") + + if ! git cherry-pick -m 1 "$sha" 2>/dev/null; then + git cherry-pick --abort 2>/dev/null + git diff HEAD "$sha" | git apply --index 2>/dev/null \ + || die "Failed to apply merge diff for ${sha:0:7}" + GIT_AUTHOR_NAME="$author_name" GIT_AUTHOR_EMAIL="$author_email" \ + GIT_AUTHOR_DATE="$author_date" \ + git commit -m "$msg" || die "Failed to commit merge ${sha:0:7}" + fi + log "INFO" "Squashed merge commit ${sha:0:7}: $(echo "$msg" | head -1)" + fi + done + + push_ref="HEAD" + linearized=true + + git push -o "base=${mono_branch}" "$(josh_auth_url)" \ + "${push_ref}:refs/heads/${staging_branch}" \ + || { log "ERROR" "Josh rejected linearized push — check josh-proxy logs"; echo "josh-rejected"; return; } + fi + + log "INFO" "Pushed to staging branch via josh: ${staging_branch}${linearized:+ (linearized)}" + + # 6. Create PR on monorepo (NEVER direct push) + local pr_body + local linearize_note="" + if [ "$linearized" = true ]; then + linearize_note=" +> **Note:** Merge commits were squashed during sync because the subrepo +> history contained merges that josh-proxy cannot map (see ADR-011). +> Regular commits are preserved individually." + fi + pr_body="## Subrepo changes New commits from subrepo \`${subrepo_branch}\`: \`\`\` ${human_commits} \`\`\` - +${linearize_note} **Review checklist:** - [ ] Changes scoped to synced subfolder - [ ] No leaked credentials or environment-specific config - [ ] CI passes" - create_pr "${MONOREPO_API}" "${GITEA_TOKEN}" \ - "$mono_branch" "$staging_branch" \ - "[Subrepo Sync] ${subrepo_branch} → ${mono_branch}" \ - "$pr_body" \ - || die "Failed to create PR on monorepo (check GITEA_TOKEN)" + create_pr "${MONOREPO_API}" "${GITEA_TOKEN}" \ + "$mono_branch" "$staging_branch" \ + "[Subrepo Sync] ${subrepo_branch} → ${mono_branch}" \ + "$pr_body" \ + || die "Failed to create PR on monorepo (check GITEA_TOKEN)" - log "INFO" "Reverse sync PR created on monorepo" - echo "pr-created" - else - log "ERROR" "Josh rejected push — check josh-proxy logs" - echo "josh-rejected" - fi + log "INFO" "Reverse sync PR created on monorepo" + echo "pr-created" } # ─── Initial Import: Subrepo → Monorepo (first time) ───────────────