From 5035bfc11704306891fa080f4597220e993a5276 Mon Sep 17 00:00:00 2001 From: Julian Cuni Date: Sat, 2 May 2026 10:58:29 +0200 Subject: [PATCH] Strip ghost-collection entries from snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third CI dry-run failure: schema-apply tried to "Create migrations_applied" and "Create positions" as Directus collections — both already exist as raw tables created by db-init pre-schema. The conflict halts schema-apply on a fresh CI DB. Why these end up in the snapshot at all: `directus schema snapshot` auto-discovers every table in the public schema, including ones owned by db-init (positions hypertable, migrations_applied guard). It registers them as ghost entries with no fields and no relations — just enough metadata to make Directus aware of the table. In local dev this never tripped because the tables existed BEFORE the snapshot ran, and any subsequent apply was a no-op against directus_collections which already had matching ghost rows. On a fresh CI DB the order is: 1. db-init pre-schema → creates the tables 2. bootstrap → installs Directus system tables (NOT the ghosts) 3. schema-apply → tries to "Create" the ghosts → conflict → fail Fixes: - snapshots/schema.yaml: stripped the migrations_applied and positions entries (24 lines each) from the collections: section. The user collections remain untouched. - scripts/schema-snapshot.sh: post-process step that filters the same ghost names from every future snapshot capture. Awk-based, applied after `docker compose cp` writes the file out. The ghost list is a bash array near the top of the new step — add to it when introducing more db-init-only tables. Snapshot is now 105 KB → ~103 KB. The user collections, fields, and relations are unchanged. positions and migrations_applied stay as raw Postgres tables managed by db-init/, never registered in directus_collections, never shown in the admin UI. That matches the schema-as-code split: Directus owns user collections; db-init owns the positions hypertable and the runner's guard table. Three CI iterations to get the boot pipeline right (port collision → ordering → ghost entries). The dry-run gate has now caught three distinct failure modes that would have damaged stage if pushed unguarded. --- scripts/schema-snapshot.sh | 41 +++++++++++++++++++++++++++++++- snapshots/schema.yaml | 48 -------------------------------------- 2 files changed, 40 insertions(+), 49 deletions(-) diff --git a/scripts/schema-snapshot.sh b/scripts/schema-snapshot.sh index 0bf6998..5911724 100755 --- a/scripts/schema-snapshot.sh +++ b/scripts/schema-snapshot.sh @@ -152,7 +152,46 @@ if [[ "${copy_exit}" -ne 0 ]]; then fi # ----------------------------------------------------------------------------- -# Step 5 — Report success +# Step 5 — Strip ghost-collection entries +# +# Directus's `schema snapshot` auto-discovers every table in the public schema +# and registers it in the snapshot YAML, regardless of whether the table is +# Directus-managed. This includes db-init-owned tables (positions hypertable, +# migrations_applied guard table) which we intentionally do NOT want Directus +# to manage. +# +# On a fresh CI Postgres, db-init creates these tables before schema-apply +# runs. If the snapshot includes them, schema-apply tries to "Create" them +# again as Directus collections — fails with "Invalid payload. Collection +# X already exists" because the underlying table already exists from db-init. +# +# Filter them out post-snapshot. Only the `collections:` section is affected +# (these tables have no fields/relations registered in directus_fields / +# directus_relations, so they only appear at the top of the YAML). +# +# Add new ghost names to this list when introducing more db-init-only tables. +# ----------------------------------------------------------------------------- + +GHOST_COLLECTIONS=( "migrations_applied" "positions" ) + +log_info "stripping ghost-collection entries from snapshot" + +for ghost in "${GHOST_COLLECTIONS[@]}"; do + # awk pattern: skip the ` - collection: ` line and all its indented + # children (meta:, schema:, etc. — 4-space indent) until the next sibling + # ` - ` or top-level section header. + awk -v ghost="${ghost}" ' + BEGIN { skip = 0 } + $0 == " - collection: " ghost { skip = 1; next } + skip && /^ - / { skip = 0 } + skip && /^[^ ]/ { skip = 0 } + !skip { print } + ' "${HOST_SNAPSHOT_PATH}" > "${HOST_SNAPSHOT_PATH}.tmp" \ + && mv "${HOST_SNAPSHOT_PATH}.tmp" "${HOST_SNAPSHOT_PATH}" +done + +# ----------------------------------------------------------------------------- +# Step 6 — Report success # ----------------------------------------------------------------------------- # Compute the size of the written file for the one-line success log. diff --git a/snapshots/schema.yaml b/snapshots/schema.yaml index 65aaaaa..a614861 100644 --- a/snapshots/schema.yaml +++ b/snapshots/schema.yaml @@ -158,30 +158,6 @@ collections: versioning: false schema: name: events - - collection: migrations_applied - meta: - accountability: all - archive_app_filter: true - archive_field: null - archive_value: null - collapse: open - collection: migrations_applied - color: null - display_template: null - group: null - hidden: false - icon: null - item_duplication_fields: null - note: null - preview_url: null - singleton: false - sort: null - sort_field: null - translations: null - unarchive_value: null - versioning: false - schema: - name: migrations_applied - collection: organization_devices meta: accountability: all @@ -282,30 +258,6 @@ collections: versioning: false schema: name: organizations - - collection: positions - meta: - accountability: all - archive_app_filter: true - archive_field: null - archive_value: null - collapse: open - collection: positions - color: null - display_template: null - group: null - hidden: false - icon: null - item_duplication_fields: null - note: null - preview_url: null - singleton: false - sort: null - sort_field: null - translations: null - unarchive_value: null - versioning: false - schema: - name: positions - collection: vehicles meta: accountability: all