Updates and maintenance

Architecture

The Swiss AI Hub separates core platform components from customer-specific code. The core platform (this repository) contains shared foundation components like API, Web, Dagster, and Bot. Customer repositories contain custom agents, pipelines, and processes. Both use independent semantic versioning and can be updated separately.

Customer code pins to a specific core version through pyproject.toml:

[project.dependencies]
swiss-ai-hub = { git = "https://github.com/bbvch-ai/aihub-core.git", tag = "v1.2.3" }

This means core updates don't automatically affect customer deployments. Customers control when they adopt new core versions.

---

Versioning

The core platform uses semantic versioning:

• Major (X.0.0): Breaking changes and architectural updates
• Minor (0.X.0): New features, backward-compatible changes
• Patch (0.0.X): Bug fixes and security patches

Three version tags are available:

Tag	Description	Stability
latest	Latest stable release	High
nightly	Latest development build	Medium
v1.2.3	Specific version tag	Highest

Customer code uses its own independent version numbers.

Release process

When a PR merges to main with a version label (major, minor, or patch), CI/CD computes the new version, creates a Git tag, and builds all affected services. Docker images are published to ghcr.io/bbvch-ai/aihub-core/* with the version tag. A changelog is generated automatically.

Each release also publishes self-contained deployment bundles as GitHub Release assets:

• swissaihub-<version>.tar.gz — CPU-only deployment bundle
• swissaihub-<version>-gpu.tar.gz — GPU-enabled deployment bundle

These bundles contain everything needed to deploy: docker-compose.yml with version-pinned image tags, all service configuration files, an .env.template with placeholder secrets, and a setup-env.sh script that generates a .env file with cryptographically secure random values for all passwords, tokens, and signing keys.

Example core images:

ghcr.io/bbvch-ai/aihub-core/api:v1.2.3
ghcr.io/bbvch-ai/aihub-core/dagster:v1.2.3
ghcr.io/bbvch-ai/aihub-core/web:v1.2.3

Customer code follows the same CI/CD pattern:

ghcr.io/bbvch-ai/aihub-core-<customer>/agent:v1.2.3
ghcr.io/bbvch-ai/aihub-core-<customer>/pipeline:v1.2.3

Browse all releases at github.com/bbvch-ai/aihub-core/releases.

---

Updates

Core platform updates

Download the new release bundle from GitHub Releases and extract it alongside your current deployment:

# Download the new version
VERSION="v1.3.0"
curl -L "https://github.com/bbvch-ai/aihub-core/releases/download/${VERSION}/swissaihub-${VERSION}.tar.gz" \
  | tar -xz -C /tmp/swissaihub-update

# Copy your existing .env into the new bundle
cp .env /tmp/swissaihub-update/.env

# Review any new environment variables added in the release
diff <(grep -oP '^[A-Z_]+=' .env.template) <(grep -oP '^[A-Z_]+=' /tmp/swissaihub-update/.env.template)

# Replace the deployment files
cp -r /tmp/swissaihub-update/* .

# Pull new images and restart
docker compose pull
docker compose up -d

Backward-compatible updates (patch and minor versions) only require pulling new images and restarting. The release bundle's docker-compose.yml already references the correct version-pinned image tags. Customer code continues running unchanged.

Major core updates with breaking changes require coordinated updates. Customer code must be updated to work with the new core version. Both core and customer code are updated together during a maintenance window.

New storage buckets

seaweedfs-init creates every missing SeaweedFS bucket each time it runs, and docker compose up -d starts it again on every run, so the update above creates the buckets a release adds. Only an update that starts selected services, such as docker compose up -d <service> or --no-deps, skips it. Then run it yourself:

docker compose up seaweedfs-init

A missing bucket fails at runtime, not at startup. Without the parse-cache bucket, every MinerU document parse fails with NoSuchBucket. Office documents are still parsed, only without the cache, so every read converts them again and logs a warning. The bucket caches parsed documents by content hash, and entries expire after 7 days. An expired entry is parsed again, so nothing needs a backup.

The sandbox-files bucket holds the one-way mirror of every user's code-sandbox home (the files behind My Files). It is created the same way, and needs SANDBOX_MIRROR_S3_ACCESS_KEY, SANDBOX_MIRROR_S3_SECRET_KEY and SANDBOX_MIRROR_INTERVAL_SECONDS in your .env (the review step above shows them). The mirror uses an S3 identity limited to this bucket, so recreate seaweedfs-s3 for the identity to take effect. Files deleted or overwritten in the sandbox are kept under .deleted/ for 7 days.

Bucket expiry

Four locations expire on their own: dagster after 1 day, and parse-cache, sandbox-files/.deleted/ and agent-files after 7 days. The one-shot seaweedfs-ttl-init service writes these as SeaweedFS filer TTL rules each time it runs, and starting seaweedfs-init starts it too, so docker compose up seaweedfs-init applies them as well. Its log lists every rule it verified:

docker compose logs seaweedfs-ttl-init

If a rule could not be written, the service shows as Exited (1) and its log names the missing rule. The rest of the stack still starts, but objects written until the next successful run never expire. Rerun it once the filer is reachable:

docker compose up seaweedfs-ttl-init

An object's expiry is fixed when it is written. Changing a rule affects new objects only.

WARNING

Never apply an S3 lifecycle configuration (put-bucket-lifecycle-configuration) to these buckets. Since SeaweedFS 4.32, lifecycle configurations expire objects only through an admin and worker service the platform does not run, and since 4.24 applying one deletes the TTL rule SeaweedFS 4.01 created for that bucket, so nothing written afterwards expires. See ADR 2026_10_27_bucket_expiry_via_filer_ttl.

SeaweedFS master state

The master keeps the cluster's identity in its data directory. If that directory is lost, or restored from a different point in time than the volume data, the master stops creating volumes as soon as a volume server reports the identity it knew, and writes fail once the existing volumes are full. After checking that no data is missing, resume volume creation:

docker compose exec seaweedfs-master sh -c 'echo volume.growth.resume | weed shell -master=localhost:9333'

Snapshot all seaweedfs-* volumes and etcd together before upgrading SeaweedFS, and restore them together.

Customer code updates

Customer code can be updated independently when the core version pin remains unchanged. Update the customer image tags in docker-compose.yml, pull the new images, and restart the customer services.

When customer code adopts a new core version, update the core version pin in pyproject.toml, rebuild the customer images, then deploy both core and customer updates together.

---

Rollbacks

VM snapshots

VM snapshots capture the entire system state. Rollback restores the complete VM from a pre-update snapshot, returning all services to their previous state at once.

Version tags

If data remains compatible with the previous version, rollback by downloading the previous release bundle, restoring your .env file, and restarting:

# Download the previous version's bundle
PREVIOUS="v1.2.3"
curl -L "https://github.com/bbvch-ai/aihub-core/releases/download/${PREVIOUS}/swissaihub-${PREVIOUS}.tar.gz" \
  | tar -xz -C /tmp/swissaihub-rollback

# Restore previous compose and configs, keep your .env
cp .env /tmp/swissaihub-rollback/.env
cp -r /tmp/swissaihub-rollback/* .

# Roll back
docker compose pull
docker compose up -d

Core and customer code can be rolled back independently if they were updated separately. If both were updated together, roll back core first, then customer code.

Services with forward-only migrations cannot be rolled back this way

"If data remains compatible" is the operative condition, and it does not hold for every service. Langfuse applies forward-only Postgres and ClickHouse migrations on startup, so once an update has run them, downloading the previous bundle restores the older binaries against the newer schema. There is no supported downgrade path.

Restoring a pre-update backup is not a rollback either: the restore validates only that the expected artifacts are present, carries no schema-version stamp, and reverts Postgres and ClickHouse independently with no cross-consistency check and no post-restore migration step.

Before updating a release that bumps Langfuse, take a backup you are willing to treat as one-way, and plan forward recovery rather than rollback.

---

Compatibility

Customer code pins to specific core versions to maintain stability. A compatibility matrix tracks which customer versions work with which core versions:

Customer Version	Core Version	Status	Notes
v1.0.0	v0.1.2	Legacy	End of life
v1.1.0	v1.2.3	Supported	Current production
v1.2.0	v1.2.3	Supported	Latest features
v2.0.0	v2.3.4	Testing	Next major release

Staging environments should match production infrastructure and use representative datasets for testing compatibility before production updates.

---

Monitoring

The observability stack includes Langfuse for AI-specific tracing, OpenTelemetry for distributed tracing, and optional SigNoz Cloud for external metrics and logs. Monitor core services (API, Web, Dagster) and customer services (agents, pipelines, processes) during and after updates.

---

Continuous Postgres maintenance

The platform includes automated database maintenance that operators do not need to schedule manually. Two jobs run in the same Dagster instance as backup (UI at http://localhost:3004) and keep the dagster database bounded over time: weekly dagster_cleanup_job prunes verbose log entries and transient framework events from event_logs, and monthly postgres_repack_job reclaims disk pages via pg_repack. Both are mutually exclusive with backup via Dagster's run-coordinator tag concurrency.

This is what keeps a 5-year-old deployment's Postgres footprint comparable to a 3-month-old one. See Backup and Recovery for retention windows and the MAINTENANCE_DISABLED kill switch.

---

Related documentation

• Deployment Options - Per-instance architecture
• Multi-tenancy - Logical separation within instances
• Backup and Recovery - Backup strategies
• Core Components - Component dependencies