pe/josh-sync
2
0
Fork 0
Code Pull Requests Actions Packages Releases Activity

Import josh-sync from subrepo/main

Browse Source 
Sync-Origin: import/josh-sync/20260528-041502
This commit is contained in:
sync-bot
2026-05-28 04:15:02 +01:00
commit b35703d271
47 changed files with 6418 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
docs/adr/001-josh-proxy-for-sync.md Normal file
View File

@@ -0,0 +1,42 @@
# ADR-001: Josh-proxy for Bidirectional Sync
**Status:** Accepted
**Date:** 2026-01
## Context
We need bidirectional sync between a monorepo and N external subrepos. Each subrepo corresponds to a subfolder in the monorepo. Developers on both sides should see a clean, complete git history — not synthetic commits or squashed blobs.
### Alternatives considered
1. **git subtree**: Built into git. `git subtree split` extracts a subfolder into a standalone repo. However, subtree split rewrites history on every run (O(n) on total commits), creating new SHAs each time. Bidirectional sync requires manual `subtree merge` with conflict-prone history grafting. No transport-layer filtering — all content must be fetched.
2. **git submodule**: Tracks external repos via `.gitmodules` pointer commits. Does not provide content-level integration — monorepo commits don't contain subrepo files directly. Developers must run `git submodule update`. Bidirectional sync is not a supported workflow.
3. **Custom diff-and-patch scripts**: Compute diffs between monorepo subfolder and subrepo, apply patches in both directions. Fragile with renames, binary files, and merge conflicts. Loses authorship and commit granularity.
4. **josh-proxy**: A git proxy that computes filtered views of repositories in real-time. Clients `git clone` through josh and receive a repo containing only the specified subfolder, with history rewritten to match. Josh maintains a persistent SHA mapping, so the same monorepo commit always produces the same filtered SHA. Bidirectional: pushing back through josh maps filtered commits to monorepo commits.
## Decision
Use josh-proxy as the transport layer for all sync operations.
## Consequences
**Positive:**
- Clean git history in both directions — no synthetic commits
- Deterministic SHA mapping — same monorepo state always produces same filtered SHA
- Bidirectional by design — push through josh maps back to monorepo
- Transport-layer filtering — content exclusion happens at clone/push time, not via generated files
- Supports any git hosting platform (Gitea, GitHub, GitLab) since it's a proxy
**Negative:**
- Requires running a josh-proxy instance (operational overhead)
- Josh-proxy is a Rust project with a smaller community than git-native tools
- Proxy must have network access to the monorepo's git server
- Josh's SHA mapping is opaque — debugging requires understanding josh internals
- First-parent traversal behavior must be respected in merge commits (see ADR-008)
**Risks:**
- Josh-proxy downtime blocks all sync operations
- Josh-proxy bugs could corrupt history mapping (mitigated by force-with-lease on forward, always-PR on reverse)

50
docs/adr/002-state-on-orphan-branch.md Normal file
View File

@@ -0,0 +1,50 @@
# ADR-002: State Storage on Orphan Git Branch
**Status:** Accepted
**Date:** 2026-01
## Context
Josh-sync needs persistent state to track what has already been synced (last-synced commit SHAs, timestamps, status). This prevents re-syncing unchanged content and enables incremental operation. The state must survive CI runner teardown — runners are ephemeral containers.
### Alternatives considered
1. **File in the repo**: Commit a state JSON file to the monorepo. Every sync run creates a commit, polluting history. Race conditions when multiple sync jobs run concurrently.
2. **External database/KV store**: Redis, SQLite, or a cloud KV service. Adds an infrastructure dependency. Credentials and connectivity to manage.
3. **CI artifacts/cache**: Platform-specific (GitHub Actions cache, Gitea cache). Not portable across CI platforms. Expiry policies vary.
4. **Orphan git branch**: A branch with no parent relationship to the main history. Stores JSON files in a simple `<target>/<branch>.json` layout. Pushed to origin, so it survives runner teardown. No external dependencies — uses git itself.
## Decision
Store sync state as JSON files on an orphan branch (`josh-sync-state`) in the monorepo.
### Storage layout
```
origin/josh-sync-state/
<target>/<branch>.json # sync state per target/branch
<target>/onboard.json # onboard workflow state (v1.1+)
```
### Implementation
- `read_state()`: `git fetch origin josh-sync-state && git show origin/josh-sync-state:<key>.json`
- `write_state()`: Uses `git worktree` to check out the orphan branch in a temp directory, writes JSON, commits, and pushes. This avoids touching the main working tree.
## Consequences
**Positive:**
- Zero external dependencies — only git
- Portable across CI platforms (Gitea Actions, GitHub Actions, local)
- Human-readable JSON files — easy to inspect and debug
- Atomic updates via git commit + push
- Natural namespacing via directory structure
**Negative:**
- Concurrent writes can race (mitigated by concurrency groups in CI workflows)
- `git worktree` adds complexity to the write path
- State branch appears in `git branch -a` output (minor clutter)
- Push failures on the state branch are non-fatal (logged as warning, sync still succeeds)

33
docs/adr/003-force-with-lease-forward.md Normal file
View File

@@ -0,0 +1,33 @@
# ADR-003: Force-with-Lease for Forward Sync
**Status:** Accepted
**Date:** 2026-01
## Context
Forward sync pushes monorepo changes to the subrepo. If someone pushes directly to the subrepo between when josh-sync reads its HEAD and when josh-sync pushes, a naive `git push` would overwrite their work. A `git push --force` would be worse — it would silently destroy concurrent changes.
## Decision
Use `git push --force-with-lease=refs/heads/<branch>:<expected-sha>` for all forward sync pushes. The expected SHA is recorded at the start of the sync operation (the "lease").
### How it works
1. Record subrepo HEAD SHA before any operations: `subrepo_sha=$(subrepo_ls_remote "$branch")`
2. Perform merge of monorepo changes onto subrepo state
3. Push with explicit lease: `--force-with-lease=refs/heads/main:<subrepo_sha>`
4. If the subrepo HEAD changed since step 1, git rejects the push
5. Josh-sync reports `lease-rejected` and retries on the next run
## Consequences
**Positive:**
- Never overwrites concurrent changes — git atomically checks the expected SHA
- Explicit SHA lease (not just "current tracking ref") prevents stale-ref bugs
- Failed leases are retried on the next sync run — no data loss, just delay
- Works correctly with josh-proxy's SHA mapping
**Negative:**
- Lease-rejected means the sync run did work that gets discarded (clone, merge, etc.)
- Persistent lease failures indicate a concurrent push pattern that needs investigation
- Requires the `--force-with-lease` flag with explicit SHA — the shorthand form (`--force-with-lease` without `=`) is unsafe because it uses the local tracking ref, which may be stale

41
docs/adr/004-always-pr-reverse.md Normal file
View File

@@ -0,0 +1,41 @@
# ADR-004: Always-PR Policy for Reverse Sync
**Status:** Accepted
**Date:** 2026-01
## Context
Reverse sync brings subrepo changes back into the monorepo. The monorepo is the source of truth and typically has CI checks, code review requirements, and branch protection rules. Pushing directly to the monorepo's main branch would bypass these safeguards.
### Alternatives considered
1. **Direct push**: Fast, but bypasses all review and CI. A bad subrepo commit could break the entire monorepo with no review gate.
2. **Always create a PR**: Pushes to a staging branch (`auto-sync/subrepo-<branch>-<timestamp>`), then creates a PR via API. Humans review and merge.
3. **Configurable per-target**: Let users choose direct push vs PR. Adds complexity and a dangerous default.
## Decision
Reverse sync always creates a PR on the monorepo. Never pushes directly to the target branch.
### Implementation
1. Push subrepo HEAD through josh-proxy to a staging branch: `git push -o "base=main" josh://... HEAD:refs/heads/auto-sync/subrepo-main-<ts>`
2. Create PR via Gitea/GitHub API targeting the monorepo's main branch
3. PR includes a review checklist: scoped to subfolder, no leaked credentials, CI passes
The `-o "base=main"` option tells josh-proxy which monorepo branch to map the push against.
## Consequences
**Positive:**
- All monorepo changes go through review — consistent with team workflow
- CI runs on the PR branch before merge
- Bad subrepo changes are caught before they affect the monorepo
- Audit trail via PR history
**Negative:**
- Reverse sync is not instant — requires human action to merge the PR
- Stale PRs accumulate if subrepo changes frequently but PRs aren't merged promptly
- Adds API dependency (needs token with PR creation scope)

52
docs/adr/005-git-trailer-loop-prevention.md Normal file
View File

@@ -0,0 +1,52 @@
# ADR-005: Git Trailer for Loop Prevention
**Status:** Accepted
**Date:** 2026-01
## Context
Bidirectional sync creates an infinite loop risk: forward sync pushes commit A to the subrepo, reverse sync sees commit A as "new" and creates a PR back to the monorepo, forward sync sees the merged PR as "new" and pushes again, etc.
### Alternatives considered
1. **SHA tracking only**: Compare SHAs to skip already-synced content. Breaks when josh-proxy rewrites SHAs (which it always does for filtered views). The monorepo commit SHA and the filtered/subrepo commit SHA are never the same.
2. **Commit message prefix**: Add `[sync]` to bot commit messages. Fragile — humans might use the same prefix. Requires string matching on message content.
3. **Git trailer**: A structured key-value pair in the commit message body (after a blank line), following the `git interpret-trailers` convention. Format: `Key: value`. Machine-parseable, unlikely to be used by humans, and supported by `git log --grep`.
## Decision
All bot commits include a git trailer with a configurable key (default: `Josh-Sync-Origin`). Both sync directions filter out commits containing this trailer.
### Format
```
Sync from monorepo 2026-02-12T10:30:00Z
Josh-Sync-Origin: forward/main/2026-02-12T10:30:00Z
```
The trailer value encodes: direction, branch, and timestamp. This aids debugging but is not parsed by the loop filter — only the trailer key presence matters.
### Filtering
- **Reverse sync**: `git log --invert-grep --grep="^${BOT_TRAILER}:"` excludes all commits with the trailer
- **CI loop guard**: The composite action checks if HEAD commit has the trailer before running sync at all
### Configuration
The trailer key is set in `.josh-sync.yml` under `bot.trailer`. This allows multiple josh-sync instances (with different bots) to operate on the same repos without interfering.
## Consequences
**Positive:**
- Reliable loop prevention — trailer is part of the immutable commit object
- Configurable key avoids conflicts between multiple sync bots
- Human-readable — `git log` shows the trailer in commit messages
- CI loop guard prevents unnecessary sync runs entirely
**Negative:**
- Commits with manually-added trailers matching the key would be incorrectly filtered
- Trailer must be in the commit body (after blank line), not the subject line
- Squash-and-merge on PRs may lose the trailer if the platform doesn't preserve commit message body

55
docs/adr/006-inline-exclude-filter.md Normal file
View File

@@ -0,0 +1,55 @@
# ADR-006: Inline Exclude in Josh-Proxy URL
**Status:** Accepted
**Date:** 2026-02
## Context
Some files in a monorepo subfolder should not appear in the subrepo (e.g., monorepo-specific CI configs, internal tooling, secrets templates). We need a mechanism to exclude these files from sync.
### Alternatives considered
1. **`.josh-sync-exclude` file committed to the repo**: A gitignore-style file listing patterns. Requires generating and committing a file. Changes to the exclude list create commits. The file itself would need to be excluded from the subrepo (circular dependency).
2. **Post-clone file deletion**: Clone through josh, then `rm -rf` excluded paths before pushing. Fragile — deletions create diff noise. Doesn't work for reverse sync (excluded files would appear as "deleted" in the subrepo).
3. **Josh `:exclude` filter inline in the URL**: Josh-proxy supports `:exclude[::pattern1,::pattern2]` appended to the filter path. The exclusion happens at the transport layer — git objects for excluded files are never transferred. Works identically for clone (forward) and push (reverse).
4. **Separate josh filter file**: Generate a josh filter expression and store it somewhere. Adds state management complexity.
## Decision
Embed exclusion patterns inline in the josh-proxy URL using josh's native `:exclude` syntax. The `exclude` config field in `.josh-sync.yml` is transformed at config parse time into the josh filter string.
### Example
Config:
```yaml
exclude:
- ".monorepo/"
- "**/internal/"
```
Produces josh filter:
```
:/services/billing:exclude[::.monorepo/,::**/internal/]
```
### Implementation
The `parse_config()` function in `lib/config.sh` uses jq to conditionally append `:exclude[...]` to the josh filter when the `exclude` array is non-empty. The enriched filter is stored in `JOSH_SYNC_TARGETS` JSON and used everywhere via `$JOSH_FILTER`.
## Consequences
**Positive:**
- Zero committed files — exclusion is purely in the URL
- Transport-layer filtering — excluded content never leaves the git server
- Works identically for forward sync (clone), reverse sync (push), and reset
- Tree comparison (`skip` detection) works correctly since excluded files aren't in the filtered view
- Standard josh syntax — no custom invention
**Negative:**
- Josh's `:exclude` pattern syntax is limited (no negation, no regex — only glob-style patterns with `::` prefix)
- Long exclude lists make the URL unwieldy (though this is cosmetic — git handles long URLs fine)
- Changing the exclude list changes the josh filter, which changes all filtered SHAs (see ADR-007 for how this is handled)
- Debugging requires understanding josh's filter composition syntax

53
docs/adr/007-reconciliation-merge.md Normal file
View File

@@ -0,0 +1,53 @@
# ADR-007: Reconciliation Merge for Filter Changes
**Status:** Accepted
**Date:** 2026-02
## Context
When the josh filter changes (e.g., adding exclude patterns), josh-proxy recomputes the entire filtered history with new SHAs. The subrepo's existing history (based on the old filter) shares no common ancestor with the new filtered history. A naive forward sync would see "unrelated histories" and fail.
### Alternatives considered
1. **Force-push to subrepo**: Replace subrepo history with the new filtered view (same as `josh-sync reset`). Destructive — all local clones become invalid, open PRs are orphaned, developers must re-clone.
2. **Cherry-pick new commits**: Identify commits that exist in the new filtered history but not the old, cherry-pick them onto the subrepo. Complex — the "same" commit has different SHAs in old vs new filtered history. No reliable way to match them.
3. **Reconciliation merge commit**: Create a merge commit on the subrepo that has both the new filtered HEAD and the old subrepo HEAD as parents, using the new filtered tree. This establishes shared ancestry without rewriting history.
## Decision
When josh-sync detects a filter change (stored filter in state differs from current `$JOSH_FILTER`), create a reconciliation merge commit using `git commit-tree`.
### How it works
1. Clone subrepo (has old history)
2. Fetch josh-proxy filtered view (has new history)
3. If trees are identical → skip (filter change had no effect on content)
4. Create merge commit: `git commit-tree <josh-tree> -p <josh-head> -p <subrepo-head>`
5. Push with `--force-with-lease`
The merge commit uses the josh-filtered tree (new content) and has two parents:
- **Parent 1**: josh-filtered HEAD (new filter history) — must be first (see ADR-008)
- **Parent 2**: subrepo HEAD (old filter history) — preserves old history as a side branch
### Detection
Filter change is detected by comparing the stored `josh_filter` in sync state with the current `$JOSH_FILTER`. For pre-v1.2 state (no filter stored), the old filter is derived as `:/<subfolder>`.
As a reactive fallback, `forward_sync()` also detects unrelated histories via `git merge-base` and falls back to reconciliation.
## Consequences
**Positive:**
- Non-destructive — old history is preserved as parent 2 of the merge
- Developers don't need to re-clone the subrepo
- Open PRs on the subrepo remain valid (they're based on commits that are still ancestors)
- Automatic — no manual intervention needed when changing exclude patterns
- Force-with-lease protects against concurrent changes during reconciliation
**Negative:**
- The merge commit is synthetic (created by bot, not a real merge of concurrent work)
- Parent ordering is critical — wrong order breaks josh's reverse mapping (see ADR-008)
- The reconciliation merge contains a bot trailer, so reverse sync correctly ignores it
- If the subrepo has diverged significantly (manual commits during filter change), the reconciliation merge may produce unexpected tree content (uses josh-filtered tree unconditionally)

42
docs/adr/008-first-parent-ordering.md Normal file
View File

@@ -0,0 +1,42 @@
# ADR-008: First-Parent Ordering in Reconciliation Merges
**Status:** Accepted
**Date:** 2026-02
## Context
Josh-proxy uses **first-parent traversal** when mapping subrepo history back to the monorepo. When you push a commit through josh-proxy, josh walks the first-parent chain to find a commit it can map to a monorepo commit. If the first parent leads to unmappable history, josh cannot reconstruct the monorepo-side branch correctly.
This became critical when the reconciliation merge (ADR-007) initially had the wrong parent order: old subrepo history as parent 1, josh-filtered as parent 2. Josh followed parent 1, couldn't find any mappable commit, and created a monorepo branch containing only the subrepo subfolder content — effectively deleting 1280 files from the rest of the monorepo.
## Decision
In reconciliation merge commits, the josh-filtered HEAD **must be parent 1** (first parent). The old subrepo HEAD is parent 2.
```bash
git commit-tree "$josh_tree" \
-p "$josh_head" \ # parent 1: josh-filtered — josh follows this
-p "$subrepo_head" \ # parent 2: old history — side branch, ignored by josh
-m "..."
```
### Why this is safe
- The old subrepo HEAD (`subrepo_head`) is still an ancestor of the merge commit regardless of parent order — push succeeds either way
- `--ancestry-path` in reverse sync still follows `B → M → C` regardless of parent order (it traces all paths, not just first-parent)
- Josh follows first-parent and finds the josh-filtered commit, which maps cleanly back to the monorepo
## Consequences
**Positive:**
- Josh can map the reconciliation merge back to the monorepo correctly
- Reverse sync through josh produces correct diffs (only subrepo-scoped changes)
- `git log --first-parent` on the subrepo shows the clean josh-filtered lineage
**Negative:**
- This is a subtle invariant — future changes to merge commit creation must preserve parent order
- The constraint is undocumented in josh-proxy's own documentation (discovered empirically)
- No automated test can verify this without a running josh-proxy instance
**Lesson learned:**
Parent order in `git commit-tree -p` is not cosmetic. For tools that rely on first-parent traversal (josh-proxy, `git log --first-parent`), parent 1 must be the "mainline" that the tool should follow.

53
docs/adr/009-tree-comparison-guard.md Normal file
View File

@@ -0,0 +1,53 @@
# ADR-009: Tree Comparison as Sync Skip Guard
**Status:** Accepted
**Date:** 2026-02
## Context
Both forward and reverse sync need to detect "nothing to do" quickly. The primary mechanism is SHA comparison against stored state (last-synced SHA). However, this misses cases where:
- State is reset or lost
- Reconciliation merges change SHAs without changing content
- Multiple sync runs overlap
Additionally, reverse sync originally relied on `git log <base>..HEAD` to find new commits. After a reconciliation merge, the `..` range can leak old subrepo history through the merge's second parent, creating false positives.
## Decision
Add tree-level comparison as an early skip guard in both forward and reverse sync. Compare the git tree objects (which represent directory content, not commit history) to determine if there's actually any content difference.
### Forward sync
```bash
mono_tree=$(git rev-parse 'HEAD^{tree}')
subrepo_tree=$(git rev-parse "subrepo/${branch}^{tree}")
[ "$mono_tree" = "$subrepo_tree" ] && echo "skip"
```
### Reverse sync
```bash
subrepo_tree=$(git rev-parse "HEAD^{tree}")
josh_tree=$(git rev-parse "mono-filtered/${branch}^{tree}")
[ "$subrepo_tree" = "$josh_tree" ] && echo "skip"
```
Tree comparison happens **before** commit log analysis. If trees are identical, there is definitionally nothing to sync, regardless of what the commit history looks like.
### Combined with `--ancestry-path`
For reverse sync, even when trees differ, `git log --ancestry-path` restricts the commit range to the direct lineage between the two endpoints. This prevents old history from leaking through reconciliation merge parents.
## Consequences
**Positive:**
- Eliminates false positives from reconciliation merges (trees are identical after reconciliation)
- Fast — tree SHA comparison is O(1), no content traversal
- Correct by definition — if trees match, content is identical
- Defense in depth — works even when state tracking has gaps
**Negative:**
- Tree comparison alone doesn't tell you *which* commits are new (still need `git log` for PR descriptions)
- Adds an extra `git rev-parse` call per sync direction (negligible cost)
- Cannot detect file-mode-only changes if josh normalizes modes (theoretical edge case)

76
docs/adr/010-onboard-checkpoint-resume.md Normal file
View File

@@ -0,0 +1,76 @@
# ADR-010: Onboard Workflow with Checkpoint/Resume
**Status:** Accepted
**Date:** 2026-02
## Context
Onboarding an existing subrepo into the monorepo is a multi-step process that involves human interaction (renaming repos, merging PRs). The full flow is:
1. Prerequisites: rename existing repo, create new empty repo
2. Import: copy subrepo content into monorepo, create import PR(s)
3. Wait: human merges the import PR(s)
4. Reset: force-push josh-filtered history to the new empty repo
5. (Optional) Migrate open PRs from archived repo
Each step can fail or be interrupted. The process may span hours or days (waiting for PR review). If interrupted, restarting from scratch wastes work and can create duplicate PRs.
### Alternatives considered
1. **Single-shot script**: Run all steps in sequence. If interrupted, must restart from scratch. Duplicate PRs if import step is re-run.
2. **Manual step-by-step commands**: `import`, then manually run `reset`. Simple but error-prone — users may forget steps or run them out of order.
3. **Checkpoint/resume with persistent state**: Track the current step and intermediate results (PR numbers, reset branches) in persistent state. On re-run, resume from the last completed step.
## Decision
Implement `josh-sync onboard` as a checkpoint/resume workflow with state stored on the `josh-sync-state` branch at `<target>/onboard.json`.
### State machine
```
start → importing → waiting-for-merge → resetting → complete
```
Each transition is persisted before proceeding. Re-running `josh-sync onboard <target>` reads the current step and resumes.
### State schema
```json
{
"step": "waiting-for-merge",
"archived_api": "https://host/api/v1/repos/org/repo-archived",
"archived_url": "git@host:org/repo-archived.git",
"archived_auth": "ssh",
"import_prs": { "main": 42 },
"reset_branches": ["main"],
"migrated_prs": [
{ "old_number": 5, "new_number": 12, "title": "Fix login" }
],
"timestamp": "2026-02-10T14:30:00Z"
}
```
### Per-branch progress
Import and reset both iterate over branches. Progress is saved after each branch, so interruption mid-iteration resumes at the next unprocessed branch.
### PR migration
`josh-sync migrate-pr` is a separate command that reads onboard state (for the archived repo URL) and tracks migrated PRs. It uses `git apply --3way` for resilient patch application — the subrepo's content is identical after reset, so patches apply cleanly.
## Consequences
**Positive:**
- Safe to interrupt at any point — no duplicate work on resume
- Per-branch tracking prevents duplicate import PRs or redundant resets
- Archived repo URL stored in state — `migrate-pr` can operate independently
- `--restart` flag allows starting over if state is corrupted
- Human-friendly — prints instructions at each step
**Negative:**
- State management adds complexity (read/write onboard state, step validation)
- Interactive steps (`read -r`) are not suitable for fully automated pipelines
- Onboard state persists on the state branch even after completion (minor clutter)
- The step machine is linear — cannot skip steps or run them out of order

58
docs/adr/011-linearize-fallback-reverse.md Normal file
View File

@@ -0,0 +1,58 @@
# ADR-011: Linearize Fallback for Reverse Sync
**Status:** Accepted
**Date:** 2026-03
## Context
Josh-proxy rejects reverse sync pushes when the subrepo history contains merge commits whose parents it cannot map through the filter. This happens when:
1. A long-lived branch (e.g., `stage`) is merged into `main` via a merge commit
2. That branch contains auto-sync merge commits or criss-cross merges with `main`
3. Josh encounters the merge commit, tries to map both parents through the filter, and fails with a 500 error
This is a legitimate subrepo workflow — teams merge staging branches into main regularly. The sync tool should handle it without requiring teams to change their Git workflow.
### Alternatives considered
1. **Squash all commits into one**: Simple but destroys all commit granularity.
2. **Rewrite subrepo history**: Breaks josh's SHA mapping, forces all developers to re-clone.
3. **Linearize: cherry-pick regular commits, squash only merges**: Preserves individual commit granularity; only merge commits lose their multi-parent structure.
## Decision
When the direct push through josh-proxy fails, fall back to option 4: linearize the history by cherry-picking onto the josh-filtered base.
### How it works
1. Direct `git push` through josh-proxy is attempted first (existing behavior)
2. If it fails, create a temporary branch from `mono-filtered/<branch>`
3. Walk human commits (oldest-first, bot commits excluded) from the ancestry path:
- **Regular commits** (≤1 parent): `git cherry-pick <sha>` — preserves author, date, and message
- **Merge commits** (>1 parent): `git cherry-pick -m 1 <sha>` — applies the merge's diff relative to its first parent as a single commit
4. If cherry-pick conflicts (rare — usually due to ordering issues), fall back to `git diff | git apply` with the original author metadata
5. Push the linearized branch through josh-proxy
6. PR body includes a note explaining that merge commits were squashed
### What is preserved
- Individual non-merge commits (author, date, message, diff)
- The net effect of each merge commit (as a squashed single commit)
- The original commit list in the PR body for reference
### What is lost
- The multi-parent structure of merge commits (they become single-parent)
- The distinction between "changes introduced by the merge" vs "changes from each parent" — the merge is represented as its diff from first parent
## Consequences
**Positive:**
- Reverse sync no longer gets stuck on merge commits — handles the common case automatically
- Non-merge commits retain full granularity (no unnecessary squashing)
- Subrepo history is untouched — no rewriting, no broken sync relationship
- The PR body documents when linearization was used, so reviewers know
**Negative:**
- Merge commit semantics are lost on the monorepo side (they appear as regular commits)
- Adds complexity to the reverse sync path (two code paths: direct push and linearize fallback)

53
docs/adr/012-explicit-monorepo-url.md Normal file
View File

@@ -0,0 +1,53 @@
# ADR-012: Explicit `monorepo_url` (drop first-target-host inference)
**Status:** Accepted
**Date:** 2026-04
## Context
josh-sync v1.x had no first-class field for the monorepo's git location. To call the gitea PR API and to clone the monorepo for `initial_import` / force-push, the script derived the gitea host from the **first target's `subrepo_url`**:
```bash
# lib/config.sh (v1.x)
gitea_host=$(echo "$JOSH_SYNC_TARGETS" | jq -r '.[0].gitea_host')
export MONOREPO_API="${MONOREPO_API:-https://${gitea_host}/api/v1/repos/${MONOREPO_PATH}}"
```
This worked silently for every existing config — studio, kinanah/stores, itkan/sdlc — because in each, target 0's host happens to match the monorepo's host. The portal monorepo broke the assumption: monorepo on `code.atome.ac:atome-portal/portal`, single target `bff` on `code.itkan.io:sdlc/odoo_bff_core`. The derivation produced `https://code.itkan.io/atome-portal/portal.git` (host borrowed from the wrong target), giving a 404 on `josh-sync import`.
Two issues stack:
1. **Implicit coupling**: target ordering becomes load-bearing in a non-obvious way. The `MONOREPO_API` env var was the only escape hatch, hidden in shell init.
2. **Asymmetry**: `subrepo_url` is declared explicitly per target with full SSH/HTTPS support; the monorepo had only a `monorepo_path` and was always cloned via HTTPS+token. Local users with SSH keys for the monorepo still had to provide a token.
### Alternatives considered
1. **Keep deriving, document the env override loudly.** Cheapest. Leaves the booby trap in place — still fails closed for any future cross-host config.
2. **Add `josh.gitea_host` only.** Fixes the host issue without enabling SSH cloning of the monorepo. Half a fix, and asymmetric with how subrepos are declared.
3. **Add `josh.monorepo_url` (full URL), mirror `subrepo_url` exactly.** Removes the inference; enables SSH clone; symmetric with subrepos. Breaking — every existing config needs to add the field.
## Decision
Add a required `josh.monorepo_url` field plus an optional `josh.monorepo_auth` (default `"https"`). Mirror the `subrepo_url` parsing and `subrepo_auth_url()` helper exactly:
- `parse_config` extracts `MONOREPO_GITEA_HOST` and `MONOREPO_REPO_PATH` from `monorepo_url` using the same jq regex as `subrepo_url`.
- `MONOREPO_API` derives from `MONOREPO_GITEA_HOST` instead of `.[0].gitea_host`. The env-var override stays as a real escape hatch (custom API endpoints).
- `mono_auth_url()` in `lib/auth.sh` mirrors `subrepo_auth_url()`: SSH returns the bare URL, HTTPS injects `${BOT_USER}:${GITEA_TOKEN}@` (auto-converting `git@host:org/repo.git` to `https://host/org/repo.git` first).
- `initial_import()` and the force-push path in `lib/sync.sh` call `mono_auth_url()` instead of constructing the URL inline from `MONOREPO_API`.
The change is shipped as **v2.0.0** with `schema_version: 2` required. v1 configs are rejected with a migration pointer; configs missing `monorepo_url` are rejected with the field name and an example.
## Consequences
**Positive:**
- No more first-target-ordering coupling. Add or reorder targets freely.
- Multi-host setups (monorepo on host A, subrepo on host B) are first-class.
- SSH clone of the monorepo works without a token (the user's ssh-agent / `~/.ssh/config` resolves the key). PR creation still needs `GITEA_TOKEN`.
- Symmetric config: monorepo and subrepos both declare a URL + auth method.
**Negative:**
- Breaking change. Every existing config must add `schema_version: 2` and `josh.monorepo_url`. Mitigated by the validation `die`s pointing at the field name and changelog.
- Two related-but-distinct fields (`monorepo_path` for the josh-proxy filter; `monorepo_url` for the gitea clone) — kept separate because josh-proxy's path may not always equal the gitea path in custom setups.
**Risks:**
- Users may set `monorepo_url` to a host that doesn't match their `proxy_url`'s upstream. Detection is out of scope for this ADR; a follow-up `josh-sync preflight` check could compare hosts and warn.

56
docs/adr/013-non-destructive-adoption.md Normal file
View File

@@ -0,0 +1,56 @@
# ADR-013: Non-destructive adoption merge for existing subrepos
**Status:** Accepted
**Date:** 2026-04
**Update (2026-05):** The standalone `josh-sync adopt <target>` command described below has been folded into `josh-sync onboard <target>` as the `adopt` strategy (selectable via `--mode=adopt`, `targets[].history_lock: preserve`, or auto-detection when the subrepo has any branches). `josh-sync adopt` is retained as a thin CLI alias. The state file moved from `<target>/adopt.json` to `<target>/onboard.json` with a `.strategy` field; the legacy file is still read for backward compatibility. See the guide's Onboarding section.
## Context
Existing subrepos often already have real history, open PRs, and developer clones.
The original `import` then `reset` onboarding path establishes Josh-compatible
history by force-pushing the Josh-filtered monorepo history onto the subrepo.
That is correct for empty replacement repos, but it rewrites the active subrepo
branch and invalidates local clones.
We need a path for active subrepos where history must remain available on the
same branch and where open PR branches should stay based on the existing history.
## Decision
Add `josh-sync adopt <target>` as a separate workflow from `onboard` and `reset`.
Adoption imports the subrepo content into the monorepo, waits for the import PR
to merge, then creates one merge commit on each configured subrepo branch:
1. First parent: Josh-filtered monorepo HEAD
2. Second parent: existing subrepo HEAD
3. Tree: Josh-filtered monorepo tree
Josh follows first-parent history back to the monorepo, while Git still keeps
the old subrepo history reachable through parent 2. The push is a normal
fast-forward push from the existing subrepo HEAD to the adoption merge commit.
No force-push is used.
Before creating the adoption merge, josh-sync requires the existing subrepo tree
to match the Josh-filtered monorepo tree. If the trees differ, adoption aborts
and the user must merge the import PR or reconcile the branch contents first.
Adoption state is stored separately at `<target>/adopt.json` on the
`josh-sync-state` branch.
## Consequences
**Positive:**
- Existing subrepo history remains on the active branch.
- Existing clones can fast-forward instead of hard-resetting or re-cloning.
- Open PR branches remain based on reachable history.
- Josh-compatible ancestry is established without a destructive rewrite.
**Negative:**
- Adoption adds one synthetic merge commit to each adopted subrepo branch.
- Strict "no subrepo commit at all" adoption is impossible if the existing
subrepo branch must become connected to the Josh-filtered history without a
rewrite. A commit is needed to join the two histories.
- Tree equality is strict. If the monorepo import differs from the subrepo
content, adoption stops until the user resolves the mismatch.

21
docs/adr/README.md Normal file
View File

@@ -0,0 +1,21 @@
# Architecture Decision Records
This directory contains Architecture Decision Records (ADRs) for josh-sync. Each ADR documents a significant design decision, its context, the alternatives considered, and the rationale for the chosen approach.
## Index
| ADR | Title | Status |
|-----|-------|--------|
| [001](001-josh-proxy-for-sync.md) | Josh-proxy for bidirectional sync | Accepted |
| [002](002-state-on-orphan-branch.md) | State storage on orphan git branch | Accepted |
| [003](003-force-with-lease-forward.md) | Force-with-lease for forward sync | Accepted |
| [004](004-always-pr-reverse.md) | Always-PR policy for reverse sync | Accepted |
| [005](005-git-trailer-loop-prevention.md) | Git trailer for loop prevention | Accepted |
| [006](006-inline-exclude-filter.md) | Inline exclude in josh-proxy URL | Accepted |
| [007](007-reconciliation-merge.md) | Reconciliation merge for filter changes | Accepted |
| [008](008-first-parent-ordering.md) | First-parent ordering in reconciliation merges | Accepted |
| [009](009-tree-comparison-guard.md) | Tree comparison as sync skip guard | Accepted |
| [010](010-onboard-checkpoint-resume.md) | Onboard workflow with checkpoint/resume | Accepted |
| [011](011-linearize-fallback-reverse.md) | Linearize fallback for reverse sync | Accepted |
| [012](012-explicit-monorepo-url.md) | Explicit `monorepo_url` (drop first-target-host inference) | Accepted |
| [013](013-non-destructive-adoption.md) | Non-destructive adoption merge for existing subrepos | Accepted |

90
docs/config-reference.md Normal file
View File

@@ -0,0 +1,90 @@
# Configuration Reference
Full reference for `.josh-sync.yml` fields and environment variables.
## `.josh-sync.yml` Structure
```yaml
schema_version: 2 # required since v2.0.0
josh: # josh-proxy + monorepo settings (required)
targets: # sync targets (required, at least 1)
bot: # bot identity for sync commits (required)
```
## `schema_version`
| Type | Required | Description |
|------|----------|-------------|
| integer | Yes | Must be `2` for josh-sync v2.x configs. v1 configs are rejected; see [Changelog v2.0.0](../CHANGELOG.md#200) for migration. |
## `josh` Section
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `proxy_url` | string | Yes | Josh-proxy URL, no trailing slash. Must start with `http://` or `https://`. |
| `monorepo_path` | string | Yes | Repository path as josh-proxy sees it (e.g., `org/monorepo`). |
| `monorepo_url` | string | Yes | Monorepo git clone URL — `git@host:org/repo.git` (SSH), `ssh://...`, or `https://host/org/repo.git`. The gitea host is derived from this URL and used for the PR API and direct clones, decoupled from any target's host. |
| `monorepo_auth` | string | No (default `"https"`) | Clone auth method: `"https"` (injects `${SYNC_BOT_USER}:${SYNC_BOT_TOKEN}` into the URL) or `"ssh"` (uses the user's ssh-agent / `~/.ssh/config`). PR creation always uses `GITEA_TOKEN` regardless. |
## `targets[]` Section
Each target maps a monorepo subfolder to an external subrepo.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `name` | string | Yes | — | Unique target identifier. Used in CLI commands and state tracking. |
| `subfolder` | string | Yes | — | Monorepo subfolder path (e.g., `services/billing`). |
| `josh_filter` | string | No | `:/<subfolder>` | Josh filter expression. Auto-derived from `subfolder` if omitted. Must start with `:`. |
| `subrepo_url` | string | Yes | — | External subrepo Git URL. Supports HTTPS (`https://...`), SSH (`git@host:path`), and `ssh://` formats. |
| `subrepo_auth` | string | No | `"https"` | Auth method: `"https"` or `"ssh"`. |
| `subrepo_token_var` | string | No | `"SUBREPO_TOKEN"` | Name of the env var holding the HTTPS token for this target. |
| `subrepo_ssh_key_var` | string | No | `"SUBREPO_SSH_KEY"` | Name of the env var holding the SSH private key for this target. |
| `branches` | object | Yes | — | Branch mapping: `mono_branch: subrepo_branch`. Each key-value pair syncs those branches bidirectionally. |
| `forward_only` | string[] | No | `[]` | Branches that only sync mono → subrepo, never reverse. |
| `exclude` | string[] | No | `[]` | File/directory patterns to exclude from sync via josh `:exclude` filter. Excluded files exist only in the monorepo, never in the subrepo. See [Excluding Files](guide.md#excluding-files-from-sync). |
| `history_lock` | string | No | (auto-detect) | Onboarding strategy hint. `"preserve"` selects the non-destructive [adopt](guide.md#onboarding) strategy (keeps subrepo history via an adoption merge — ADR-013); `"rewrite"` selects the destructive reset strategy (force-pushes josh-filtered history). When omitted, `josh-sync onboard` auto-detects: subrepos with branches → adopt, empty subrepos → reset. The CLI flag `--mode={reset,adopt}` overrides this. |
## `bot` Section
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | Yes | Git author name for sync commits. |
| `email` | string | Yes | Git author email for sync commits. |
| `trailer` | string | Yes | Git trailer key for loop prevention (e.g., `Josh-Sync-Origin`). |
## Environment Variables
### Credentials
| Variable | Purpose | Default |
|----------|---------|---------|
| `SYNC_BOT_USER` | Bot's Git username | — |
| `SYNC_BOT_TOKEN` | API token for monorepo access and josh-proxy HTTPS auth | — |
| `SUBREPO_TOKEN` | HTTPS token for subrepo access | Falls back to `SYNC_BOT_TOKEN` |
| `SUBREPO_SSH_KEY` | SSH private key content (not a file path) for subrepo access | — |
### Per-target credential overrides
Set `subrepo_token_var` or `subrepo_ssh_key_var` in a target's config to read credentials from a different env var:
```yaml
targets:
- name: "auth"
subrepo_token_var: "AUTH_REPO_TOKEN" # reads from $AUTH_REPO_TOKEN
subrepo_ssh_key_var: "AUTH_SSH_KEY" # reads from $AUTH_SSH_KEY
```
**Resolution order:** per-target env var → default env var (`SUBREPO_TOKEN` / `SUBREPO_SSH_KEY`) → `SYNC_BOT_TOKEN` fallback.
### Runtime
| Variable | Purpose | Default |
|----------|---------|---------|
| `JOSH_SYNC_TARGET` | Restrict sync to a single target | All targets |
| `JOSH_SYNC_STATE_BRANCH` | Name of the orphan branch for state storage | `josh-sync-state` |
| `JOSH_SYNC_DEBUG` | Enable verbose logging (`1` to enable) | `0` |
| `MONOREPO_API` | Override the monorepo gitea PR API URL. Escape hatch for custom API endpoints; rarely needed. | `https://{monorepo_url's host}/api/v1/repos/{monorepo_path}` |
## JSON Schema
The config file can be validated against the JSON Schema at [`schema/config-schema.json`](../schema/config-schema.json).

758
docs/guide.md Normal file
View File

@@ -0,0 +1,758 @@
# Setup Guide
Step-by-step guide to setting up josh-sync for a new monorepo with existing subrepos.
## Overview
josh-sync provides bidirectional sync between a monorepo and N external subrepos via [josh-proxy](https://josh-project.github.io/josh/):
```
MONOREPO SUBREPOS
├── services/billing/ ──── forward ────► billing-repo/
├── services/auth/ (push or cron) auth-repo/
└── libs/shared/ ◄──── reverse ───── shared-lib-repo/
(cron → always PR)
via josh-proxy (filtered git views)
```
**Key safety properties:**
- Forward sync (mono → subrepo) uses `--force-with-lease` — never overwrites concurrent changes
- Reverse sync (subrepo → mono) always creates a PR — never pushes directly
- Git trailers (`Josh-Sync-Origin:`) prevent infinite sync loops
- State tracked on an orphan branch (`josh-sync-state`) — survives CI runner teardown
## Prerequisites
Before you begin, you need:
### josh-proxy instance
A running [josh-proxy](https://josh-project.github.io/josh/) that can access your monorepo's Git server. Verify connectivity:
```bash
git ls-remote https://josh.example.com/org/monorepo.git HEAD
```
### Bot account
A dedicated Git user (e.g., `josh-sync-bot`) with:
- Write access to the monorepo
- Write access to all subrepos
- Ability to create PRs on both monorepo and subrepo platforms
### Credentials
| Variable | Purpose | Required |
|----------|---------|----------|
| `SYNC_BOT_USER` | Bot's Git username | Yes |
| `SYNC_BOT_TOKEN` | API token with repo scope (monorepo + josh-proxy auth) | Yes |
| `SUBREPO_SSH_KEY` | SSH private key for subrepo access (if using SSH auth) | If SSH |
| `SUBREPO_TOKEN` | HTTPS token for subrepo access (defaults to `SYNC_BOT_TOKEN`) | No |
Per-target credential overrides are supported — see [Configuration Reference](config-reference.md).
### Tool dependencies
`bash >=4`, `git`, `curl`, `jq`, `yq` ([mikefarah/yq](https://github.com/mikefarah/yq) v4+), `openssh`, `rsync`
> The Nix flake bundles all dependencies automatically.
## Step 1: Create the Monorepo
Create a new repository on your Git server (e.g., `org/monorepo`). Create subdirectories for each subrepo you want to sync:
```bash
mkdir -p services/billing services/auth libs/shared
```
These directories will be populated during the import step. They can be empty or contain `.gitkeep` files for now.
Verify josh-proxy can see the monorepo:
```bash
git ls-remote https://josh.example.com/org/monorepo.git HEAD
```
## Step 2: Configure `.josh-sync.yml`
Create `.josh-sync.yml` at the monorepo root. Each target maps a monorepo subfolder to an external subrepo:
```yaml
schema_version: 2
josh:
proxy_url: "https://josh.example.com" # josh-proxy URL (no trailing slash)
monorepo_path: "org/monorepo" # repo path as josh-proxy sees it
monorepo_url: "git@gitea.example.com:org/monorepo.git" # monorepo git clone URL
monorepo_auth: "ssh" # optional, default "https"
targets:
- name: "billing" # unique identifier
subfolder: "services/billing" # monorepo subfolder
# josh_filter auto-derived as ":/services/billing" if omitted
subrepo_url: "git@gitea.example.com:ext/billing.git"
subrepo_auth: "ssh" # "https" (default) or "ssh"
branches:
main: main # mono_branch: subrepo_branch
forward_only: []
exclude: # files excluded from subrepo (optional)
- ".monorepo/" # monorepo-only config dir
- "**/internal/" # internal dirs at any depth
- name: "auth"
subfolder: "services/auth"
subrepo_url: "https://gitea.example.com/ext/auth.git"
subrepo_auth: "https"
subrepo_token_var: "AUTH_REPO_TOKEN" # per-target credential override
branches:
main: main
develop: develop # multiple branches supported
forward_only: []
- name: "shared-lib"
subfolder: "libs/shared"
subrepo_url: "https://gitea.example.com/ext/shared-lib.git"
branches:
main: main
forward_only: [main] # one-way: mono → subrepo only
bot:
name: "josh-sync-bot"
email: "josh-sync-bot@example.com"
trailer: "Josh-Sync-Origin" # git trailer for loop prevention
```
For the full field reference, see [Configuration Reference](config-reference.md).
## Step 3: Set Up Local Dev Environment
### Option A: Nix + devenv (recommended)
**`devenv.yaml`** — declare josh-sync as a flake input:
```yaml
inputs:
nixpkgs:
url: github:cachix/devenv-nixpkgs/rolling
josh-sync:
url: git+https://your-gitea.example.com/org/josh-sync?ref=main
flake: true
```
**`devenv.nix`** — import the josh-sync module:
```nix
{ inputs, ... }:
{
imports = [ inputs.josh-sync.devenvModules.default ];
name = "my-monorepo";
# .env contains secrets, not devenv config
dotenv.disableHint = true;
}
```
**`.envrc`** — activate devenv automatically:
```bash
DEVENV_WARN_TIMEOUT=20
use devenv
```
**`.env`** — local credentials (add to `.gitignore`):
```bash
SYNC_BOT_USER=sync-bot
SYNC_BOT_TOKEN=<your-api-token>
SUBREPO_SSH_KEY="-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----"
# Per-target overrides:
# AUTH_REPO_TOKEN=<auth-specific-token>
```
### Updating josh-sync in devenv
To update to the latest version:
```bash
devenv update josh-sync
```
Or with plain Nix flakes:
```bash
nix flake lock --update-input josh-sync
```
To pin to a specific version, use a tag ref in `devenv.yaml`:
```yaml
josh-sync:
url: git+https://your-gitea.example.com/org/josh-sync?ref=refs/tags/v1.2
flake: true
```
After updating, verify the version:
```bash
josh-sync --version
```
### Option B: Manual installation
Install the required tools, then either:
- Clone the josh-sync repo and add `bin/` to your `PATH`
- Run `make build` to create a single bundled script at `dist/josh-sync`
## Step 4: Validate with Preflight
```bash
josh-sync preflight
```
This validates:
- Config syntax and required fields
- josh-proxy connectivity (via `git ls-remote` through josh)
- Subrepo connectivity and authentication
- Branch mappings
- CI workflow path coverage (checks if `.gitea/workflows/josh-sync-forward.yml` paths match target subfolders)
For a new monorepo before import, preflight may warn that subfolders don't exist yet — that's expected.
## Step 5: Onboard Existing Subrepos
`josh-sync onboard <target>` is the single entry point for connecting an existing subrepo to the monorepo. It runs interactively with checkpoint/resume and picks one of two strategies:
- **adopt** (non-destructive) — keeps the subrepo's existing history and joins it to josh-filtered history via a single adoption merge commit. No force-push. Developers fast-forward; open PR branches stay valid.
- **reset** (destructive) — force-pushes josh-filtered history onto the subrepo, replacing its prior history. Use this for empty replacement repos (typically paired with renaming the old repo to `*-archived`).
### Strategy selection
`josh-sync onboard` resolves the strategy in this order (highest precedence first):
1. **CLI flag** — `--mode=adopt` or `--mode=reset` overrides everything.
2. **Config** — `targets[].history_lock: preserve` selects adopt; `rewrite` selects reset.
3. **Auto-detect** — `git ls-remote --heads` on the subrepo. Branches present → adopt. Empty repo → reset.
The chosen strategy is logged at preflight so you can see what will happen before any side effects.
`josh-sync adopt <target>` is kept as a back-compat alias for `josh-sync onboard --mode=adopt`.
### Adopt strategy (auto-selected for active subrepos)
```bash
josh-sync onboard billing # auto-detect — picks adopt for non-empty subrepo
josh-sync onboard billing --mode=adopt # force adopt explicitly
josh-sync adopt billing # equivalent back-compat alias
```
The command will:
1. **Import** — copies current subrepo content into the monorepo and creates import PRs (one per branch)
2. **Wait for merge** — shows PR numbers and waits for you to merge them
3. **Verify trees** — requires the existing subrepo tree to match the Josh-filtered monorepo tree
4. **Adopt** — creates a merge commit on the subrepo with Josh-filtered HEAD as parent 1 and existing subrepo HEAD as parent 2
5. **Push** — pushes the adoption merge with a normal fast-forward push, never force-push
The adoption merge preserves the old subrepo history while giving Josh a first-parent path back to the monorepo. If interrupted, re-run `josh-sync onboard billing` to resume. Use `--restart` to start over. See [ADR-013](adr/013-non-destructive-adoption.md) for the design rationale.
After adoption, developers can update existing clones with a normal fast-forward:
```bash
git fetch origin
git checkout main
git merge --ff-only origin/main
```
### Reset strategy (auto-selected for empty replacement repos)
Use the reset strategy when you intentionally want to archive the old subrepo and start fresh from a new empty repo.
**Before you start:**
1. **Rename** the existing subrepo on your Git server (e.g., `stores/storefront` → `stores/storefront-archived`)
2. **Create a new empty repo** at the original path (e.g., a new `stores/storefront` with no commits)
The rename preserves the archived repo with all its history and open PRs. The new empty repo will receive josh-filtered history.
**Run onboard:**
```bash
josh-sync onboard billing # auto-detect — picks reset for empty subrepo
josh-sync onboard billing --mode=reset # force reset explicitly
```
The command will:
1. **Verify prerequisites** — checks the new empty repo is reachable, asks for the archived repo URL
2. **Import** — copies subrepo content into monorepo and creates import PRs (one per branch)
3. **Wait for merge** — shows PR numbers and waits for you to merge them
4. **Reset** — pushes josh-filtered history to the new subrepo (per-branch, with resume)
5. **Done** — prints instructions for developers and PR migration
If the process is interrupted at any point, re-run `josh-sync onboard billing` to resume from where it left off. Use `--restart` to start over.
**Migrate open PRs:**
After onboard completes, migrate PRs from the archived repo to the new one:
```bash
# Interactive — lists open PRs and lets you pick
josh-sync migrate-pr billing
# Migrate all open PRs at once
josh-sync migrate-pr billing --all
# Migrate specific PRs by number
josh-sync migrate-pr billing 5 8 12
```
PR migration works by fetching the diff from the archived repo's PR, applying it to the new repo, and creating a new PR. File content is identical after reset, so patches apply cleanly.
### Manual: import → merge → reset
Use this for scripted automation or when you intentionally want to replace subrepo history without the interactive wrapper.
> Do this **one target at a time** to keep PRs reviewable.
#### Manual: Import
```bash
josh-sync import billing
```
This:
1. Clones the monorepo directly (not through josh)
2. Clones the subrepo
3. Copies subrepo content into the monorepo subfolder via `rsync`
4. Creates a branch `auto-sync/import-billing-<timestamp>`
5. Pushes it and creates a PR on the monorepo
Review the import PR — check for leaked credentials, environment-specific config, or files that shouldn't be in the monorepo.
#### Manual: Merge the import PR
Merge the PR using your Git platform's UI. This lands the subrepo content into the monorepo's main branch.
> At this point, the monorepo has the content but the histories are disconnected. Sync will **not** work until you complete the reset step.
#### Manual: Reset
```bash
josh-sync reset billing
```
> **You do NOT need to `git pull` locally before running reset.** The reset command clones fresh from josh-proxy — it never uses your local working copy.
This:
1. Clones the monorepo through josh-proxy with the josh filter (the "filtered view")
2. Force-pushes that filtered view to the subrepo, replacing its history
This establishes **shared commit ancestry** between josh's filtered view and the subrepo. Without this, josh-proxy can't compute diffs between the two.
> **Warning:** This is a destructive force-push that replaces the subrepo's history. Back up any important branches or tags in the subrepo beforehand. Merge or close all open pull requests on the subrepo first — they will be invalidated.
After reset, **every developer with a local clone of the subrepo** must update their local copy to match the new history:
```bash
cd /path/to/local-subrepo
git fetch origin
git checkout main && git reset --hard origin/main
git checkout stage && git reset --hard origin/stage # repeat for each branch
```
Or simply delete and re-clone the subrepo. Local-only branches (not pushed to the remote) will be lost either way.
#### Manual: Repeat for each target
```
For each target:
1. josh-sync import <target>
2. Review and merge the import PR on the monorepo
3. josh-sync reset <target>
```
### Verify
After all targets are adopted or reset:
```bash
# Check all targets show state
josh-sync status
# Test forward sync — should return "skip" (trees are identical after adoption/reset)
josh-sync sync --forward --target billing
# Test reverse sync — should return "skip" (no new human commits)
josh-sync sync --reverse --target billing
```
## Step 6: Set Up CI Workflows
### Forward sync (mono → subrepo)
Create `.gitea/workflows/josh-sync-forward.yml`:
```yaml
name: "Josh Sync → Subrepo"
on:
push:
branches: [main]
paths:
# List ALL target subfolders:
- "services/billing/**"
- "services/auth/**"
- "libs/shared/**"
schedule:
- cron: "0 */6 * * *" # every 6 hours as fallback
workflow_dispatch:
inputs:
target:
description: "Target to sync (empty = detect from push or all)"
required: false
default: ""
branch:
description: "Branch to sync (empty = triggered branch or all)"
required: false
default: ""
concurrency:
group: josh-sync-fwd-${{ github.ref_name }}
cancel-in-progress: false
jobs:
sync:
runs-on: docker
container: node:20-bookworm
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2 # needed for target detection
- name: Install tools
run: |
apt-get update -qq && apt-get install -y -qq jq curl git openssh-client >/dev/null 2>&1
curl -sL "https://github.com/mikefarah/yq/releases/download/v4.44.6/yq_linux_amd64" \
-o /usr/local/bin/yq && chmod +x /usr/local/bin/yq
- name: Detect changed target
if: github.event_name == 'push'
id: detect
run: |
CHANGED=$(git diff --name-only HEAD~1 HEAD 2>/dev/null || echo "")
TARGETS=$(yq -o json '.targets' .josh-sync.yml \
| jq -r '.[] | "\(.name):\(.subfolder)"' \
| while IFS=: read -r name prefix; do
echo "$CHANGED" | grep -q "^${prefix}/" && echo "$name"
done | sort -u | paste -sd ',' -)
echo "targets=${TARGETS}" >> "$GITHUB_OUTPUT"
- uses: https://your-gitea.example.com/org/josh-sync@v1
with:
direction: forward
target: ${{ github.event.inputs.target || steps.detect.outputs.targets }}
branch: ${{ github.event.inputs.branch || github.ref_name }}
env:
SYNC_BOT_USER: ${{ secrets.SYNC_BOT_USER }}
SYNC_BOT_TOKEN: ${{ secrets.SYNC_BOT_TOKEN }}
SUBREPO_TOKEN: ${{ secrets.SUBREPO_TOKEN || secrets.SYNC_BOT_TOKEN }}
SUBREPO_SSH_KEY: ${{ secrets.SUBREPO_SSH_KEY }}
```
### Reverse sync (subrepo → mono)
Create `.gitea/workflows/josh-sync-reverse.yml`:
```yaml
name: "Josh Sync ← Subrepo"
on:
schedule:
- cron: "0 1,7,13,19 * * *" # every 6h, offset from forward
workflow_dispatch:
inputs:
target:
description: "Target to sync (empty = all)"
required: false
default: ""
branch:
description: "Branch to sync (empty = all eligible)"
required: false
default: ""
concurrency:
group: josh-sync-rev-${{ github.event.inputs.target || 'all' }}
cancel-in-progress: false
jobs:
sync:
runs-on: docker
container: node:20-bookworm
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Install tools
run: |
apt-get update -qq && apt-get install -y -qq jq curl git openssh-client >/dev/null 2>&1
curl -sL "https://github.com/mikefarah/yq/releases/download/v4.44.6/yq_linux_amd64" \
-o /usr/local/bin/yq && chmod +x /usr/local/bin/yq
- uses: https://your-gitea.example.com/org/josh-sync@v1
with:
direction: reverse
target: ${{ github.event.inputs.target || '' }}
branch: ${{ github.event.inputs.branch || '' }}
env:
SYNC_BOT_USER: ${{ secrets.SYNC_BOT_USER }}
SYNC_BOT_TOKEN: ${{ secrets.SYNC_BOT_TOKEN }}
SUBREPO_TOKEN: ${{ secrets.SUBREPO_TOKEN || secrets.SYNC_BOT_TOKEN }}
SUBREPO_SSH_KEY: ${{ secrets.SUBREPO_SSH_KEY }}
```
### Required CI secrets
| Secret | Purpose |
|--------|---------|
| `SYNC_BOT_USER` | Bot username |
| `SYNC_BOT_TOKEN` | Bot API token (monorepo access + josh-proxy auth) |
| `SUBREPO_SSH_KEY` | SSH private key for subrepo push (if using SSH auth) |
| `SUBREPO_TOKEN` | Optional separate subrepo token (defaults to `SYNC_BOT_TOKEN`) |
> **GitHub Actions note:** These examples target Gitea Actions. For GitHub Actions, change the `uses:` reference to a GitHub repo (e.g., `org/josh-sync@v1`) and `runs-on:` to a GitHub runner (e.g., `ubuntu-latest`).
## How Ongoing Sync Works
Once set up, sync runs automatically:
### Forward sync (mono → subrepo)
Triggered by pushes to target subfolders or on a cron schedule:
1. Clones the monorepo through josh-proxy (filtered view of the subfolder)
2. Fetches the subrepo branch for comparison
3. If trees are identical → skip
4. If subrepo branch doesn't exist → fresh push
5. Merges mono changes on top of subrepo state
6. If clean merge → pushes with `--force-with-lease` (protects against concurrent changes)
7. If lease rejected → retries on next run (subrepo changed during sync)
8. If merge conflict → creates a conflict PR on the subrepo
### Reverse sync (subrepo → mono)
Runs on a cron schedule (never triggered by subrepo pushes):
1. Clones the subrepo
2. Fetches the monorepo's josh-filtered view for comparison
3. Finds new human commits (filters out bot commits by checking for the `Josh-Sync-Origin:` trailer)
4. If no new human commits → skip
5. Pushes through josh-proxy to a staging branch
6. Creates a PR on the monorepo — **never pushes directly**
### Loop prevention
Bot commits include a git trailer like `Josh-Sync-Origin: forward/main/2024-02-12T10:30:00Z`. Each sync direction filters out commits with this trailer, preventing changes from bouncing back and forth. The CI action also has a loop guard that skips entirely if the HEAD commit has the trailer.
### State tracking
Sync state is stored as JSON files on an orphan branch (`josh-sync-state`), one file per target/branch. This tracks the last-synced commit SHAs and timestamps to avoid re-syncing the same changes.
## Recommended Git Workflow
Josh-proxy maps commits through a filter per-branch. It handles linear history and simple merges (short-lived feature branches) without issues. However, it **cannot map merge commits whose parents were created on the subrepo side** — because those commits were never pushed through josh and have no monorepo-side mapping.
This means cross-branch merges (e.g., `stage` → `main`) must happen on the monorepo side, where josh can filter the result cleanly.
### The rule
**The monorepo owns branch topology. The subrepo owns feature development.**
### What to do where
| Action | Where to do it | Why |
|--------|---------------|-----|
| Feature branch → `main` | **Subrepo** (PR, any merge strategy) | Short-lived branch with clean lineage — josh handles it |
| `stage` → `main` (promotion) | **Monorepo** | Cross-branch merge — forward sync propagates the result to both subrepo branches |
| `main` → `stage` (catch-up) | **Monorepo** | Same reason — avoids criss-cross merge history on subrepo |
| Hotfix to `main` | **Either side** | Single commit or small PR — works everywhere |
| Config/CI changes (monorepo-only) | **Monorepo** | Not synced to subrepo (use `exclude` for monorepo-only files) |
### What to avoid
- **Don't merge `stage` into `main` on the subrepo with a merge commit.** The merge parents include commits created on the subrepo side (forward sync merges, criss-cross merges) that josh has no mapping for. Josh rejects the push with a 500 error.
- **Don't merge `main` into `stage` on the subrepo.** Creates criss-cross merge history that causes the same josh mapping failure when `stage` is later merged back.
- **Don't rebase synced branches on the subrepo.** This rewrites commit SHAs that josh has already mapped, breaking the sync relationship.
### If you must merge cross-branch on the subrepo
Use a **squash merge**. A squash merge produces a single commit with one parent — josh can always map it. You lose the individual commit history on the target branch, but the sync goes through cleanly.
As a safety net, josh-sync automatically falls back to linearizing the history when josh rejects a push — cherry-picking regular commits individually and squashing only the problematic merge commits. See the [troubleshooting section](#josh-rejected-push-reverse-sync) and [ADR-011](adr/011-linearize-fallback-reverse.md).
## Excluding Files from Sync
Some files in the monorepo subfolder may not belong in the subrepo (e.g., monorepo-specific CI configs, internal tooling). The `exclude` config field removes these at the josh-proxy layer — excluded files never appear in the subrepo.
### Configuration
Add an `exclude` list to any target:
```yaml
targets:
- name: "billing"
subfolder: "services/billing"
subrepo_url: "git@host:org/billing.git"
exclude:
- ".monorepo/" # directory at subfolder root
- "**/internal/" # directory at any depth
- "*.secret" # files by extension
branches:
main: main
```
### How it works
When `exclude` is present, josh-sync appends an inline `:exclude` filter to the josh-proxy URL. For the example above, the josh filter becomes:
```
:/services/billing:exclude[::.monorepo/,::**/internal/,::*.secret]
```
Josh-proxy applies this filter at the transport layer — no extra files to generate or commit. This means:
- **Forward sync**: the filtered clone already excludes the files
- **Reverse sync**: pushes through josh also respect the exclusion
- **Reset**: the subrepo history never contains excluded files
- **Tree comparison**: `skip` detection works correctly (excluded files are not in the diff)
### Pattern syntax
Josh uses `::` patterns inside `:exclude[...]`:
| Pattern | Matches |
|---------|---------|
| `dir/` | Directory at subfolder root |
| `file` | File at subfolder root |
| `**/dir/` | Directory at any depth |
| `**/file` | File at any depth |
| `*.ext` | Glob pattern (single `*` only) |
### Setup
1. Add `exclude` to the target in `.josh-sync.yml`
2. Run `josh-sync preflight` to verify the filter works
3. Forward sync will now exclude the specified files
No extra files to generate or commit — the exclusion is embedded directly in the josh-proxy URL.
### Changing the exclude list
You can safely add or remove patterns from `exclude` at any time. When josh-sync detects that the filter has changed since the last sync, it automatically creates a reconciliation merge commit on the subrepo that connects the old and new histories — no manual reset or force-push required. Developers do not need to re-clone the subrepo.
## Adding a New Target
To add a new subrepo after initial setup:
1. Add the target to `.josh-sync.yml`
2. Update the forward workflow's `paths:` list to include the new subfolder
3. Commit and push
4. Onboard the target — `josh-sync onboard` auto-picks `adopt` for non-empty subrepos and `reset` for empty ones; override with `--mode={reset,adopt}` or pin via the target's `history_lock` config field.
```bash
josh-sync onboard new-target # auto-detect strategy
josh-sync onboard new-target --mode=adopt # force adopt
josh-sync onboard new-target --mode=reset # force reset
# Manual lower-level path (when scripting):
josh-sync import new-target
# merge the PR
josh-sync reset new-target
```
5. Verify with `josh-sync status`
## Troubleshooting
### "Failed to clone through josh-proxy"
- Check josh-proxy is running and accessible
- Verify `monorepo_path` matches what josh-proxy expects
- Test manually: `git ls-remote https://<user>:<token>@josh.example.com/org/repo.git:/services/app.git`
### SSH authentication failures
- `SUBREPO_SSH_KEY` must contain the actual key content, not a file path
- For per-target keys, ensure `subrepo_ssh_key_var` in config matches the env var name
- Check the key has write access to the subrepo
### "Force-with-lease rejected"
Normal: the subrepo changed while sync was running. The next sync run will pick it up. If persistent, check for another process pushing to the subrepo simultaneously.
### "Josh rejected push" (reverse sync)
Josh-proxy couldn't map the push back to the monorepo. This has two common causes:
#### Merge commits with unmappable parents
**Symptom:** Josh returns `500 Internal Server Error` with a message like:
```
rejecting merge with 2 parents:
"Merge pull request 'stage' (#30) from stage into main" (c4fa3c9...)
1) "Merge branch 'auto-sync/import-...'" (4bf8704...)
2) "Merge branch 'main' into stage" (d021654...)
```
**Cause:** The subrepo has a merge commit whose parents josh-proxy cannot trace through its filter. This typically happens when:
- A long-lived branch (e.g., `stage`) is merged into `main` via a merge commit (not squash)
- That branch contains auto-sync merge commits or other history that doesn't exist in josh's filtered view
- Someone merges `main` into a feature/staging branch and then merges it back — the criss-cross parents confuse josh's mapping
**Automatic handling:** josh-sync automatically falls back to linearizing the history when the direct push fails. Regular commits are cherry-picked individually (preserving authorship and messages), while merge commits are squashed into single commits via `cherry-pick -m 1`. The PR notes when this fallback was used. See [ADR-011](adr/011-linearize-fallback-reverse.md).
**Prevention:** In josh-synced subrepos, prefer **squash merges** when merging long-lived branches (stage, develop) into the synced branch. Squash merges produce a single commit with no merge parents, which josh can always map. Regular feature branch merges (short-lived, no auto-sync history) are usually fine.
**Manual resolution (if automatic fallback also fails):** This indicates a more fundamental history issue. Options:
1. Cherry-pick the desired changes manually onto a clean branch and push through josh
2. Run `josh-sync reset <target>` to re-establish history (destructive — all subrepo clones must re-fetch)
#### Filter or path mismatch
Josh-proxy couldn't map the push due to an incorrect filter. Check josh-proxy logs, verify the `josh_filter` or `subfolder` in `.josh-sync.yml` is correct, and ensure the subfolder exists in the monorepo.
### Import PR shows "No changes"
The subfolder already contains the same content as the subrepo. This is fine — the import is a no-op.
### Duplicate/looping commits
Verify `bot.trailer` in config matches what's in commit messages. Check the loop guard in the CI workflow is active.
### "cannot lock ref" or "expected X but got Y"
**After reset (subrepo):** The subrepo's history was replaced by force-push. Local clones still have the old history:
```bash
cd /path/to/subrepo
git fetch origin
git checkout main && git reset --hard origin/main
```
Or simply delete and re-clone.
**After import/reset cycle (monorepo):** The import and reset steps create and update branches rapidly (`auto-sync/import-*`, `josh-sync-state`). If your local clone fetched partway through, tracking refs go stale:
```bash
git remote prune origin && git pull
```
### State issues
```bash
# View current state
josh-sync state show <target> [branch]
# Reset state (forces next sync to run regardless of SHA comparison)
josh-sync state reset <target> [branch]
```