Legacy RAG Pipelines: Code Removed, Images Frozen, Names Reserved

Context

Before the Generic Document Ingestion Pipeline, a knowledge database was served by a pipeline bound to one bucket at deploy time. Two such deployments exist: default_rag_pipeline (AIHubSettings().DEFAULT_BUCKET_NAME, defaultknowledge) and shared_rag_pipeline (SHARED_BUCKET_NAME, sharedknowledge). Both are built by default_definitions, the public SDK builder for fixed-bucket pipelines.

document_ingestion_pipeline supersedes them entirely: one deployment serves every knowledge database, resolving its target per run. Nothing new is ever assigned to a legacy pipeline: their routing tokens are reserved, so no pipeline can register them and the API never offers them (see 2026_09_04_ingestors_announce_their_configuration_form). The two remaining deployments exist solely to keep ingesting into corpora that already exist.

Keeping them alive is not free. Route-per-run forked the whole Stage-2 storage surface into ~12 routed_* modules, each a twin of a non-routed counterpart differing only in how it learns the bucket. After the legacy apps stop being deployed, the non-routed set is maintained only for them and for a dev example — and the routed_ prefix names a distinction that no longer distinguishes anything.

Decision Drivers

1. A half-migration is worse than either end of it. Two parallel store-construction paths for one job, and a prefix that disambiguates from files nobody uses, cost more in confusion than the legacy pipelines are worth.
2. Existing corpora must keep working. A customer running defaultknowledge today cannot be forced into a data migration by a platform upgrade.
3. Frozen must mean frozen. Anything that implies the legacy images still receive changes — a build context, a latest tag, a code location in a stage that does not run them — is a promise the platform cannot keep.

Decision

Delete the legacy pipelines' code; keep their last published images pinned, in the stages that run them, forever.

• The code is gone. app/default_rag_pipeline, app/shared_rag_pipeline, default_definitions and the ~12 non-routed Stage-2 twins are deleted, and the routed_ prefix is dropped from the modules that outlive them. The prefix survives only on the six wrapper resources whose non-routed counterparts are still used by Stage 1.

• Removing default_definitions is a breaking SDK change, accepted deliberately rather than deprecated: it built fixed-bucket pipelines, which is the model being retired. document_ingestion_pipeline_definitions replaces it and is now the exported builder.

• The images are pinned to v0.320.1 in nightly and latest only. They carry no build or local key, so they render in no other stage; the compose template's build branch is removed, since the Dockerfile it referenced no longer exists on main. CI leaves them alone by construction: image discovery requires build: localbuild, and the latest retagging step requires a :latest tag.

  v0.320.0 is the last release that still contained the legacy source — this decision landed on main after that release was cut. v0.320.1 is that code plus the knowledge namespace teardown sensor, built once from fix/legacy-namespace-teardown — a single commit on top of the v0.320.0 tag — and pushed with no secondary tag. The freeze deliberately takes effect at the version where a frozen corpus can still have folders removed from it, because the alternative was shipping a corpus that could be added to forever and never pruned. Frozen still means frozen: v0.320.1 is the terminal version, and the branch it was built from is not maintained.

  That branch is the only ref carrying this code: it is not on main, which deleted the legacy source, and not on release/0.320, which never received the teardown sensor. The git tag v0.320.1 points at its tip so the image tag and the source have a shared anchor. Neither the branch nor the tag may be deleted — without them the running images have no locatable source.

• The Dagster workspace is guarded on the same image tags as the compose services, so a code location can no longer outlive its container. This is what actually went wrong before: the services were commented out of compose-config but left in workspace.yml.j2, giving every deployed stage two permanently-errored code locations.

• There is no migration path. A deployment that needs any change to a legacy corpus re-uploads it into a new self-service knowledge database. Re-pointing the existing buckets at document_ingestion_pipeline is not offered: the two pipelines would then both claim the same bucket, and the embedding model a corpus was ingested with is not recorded anywhere that a re-ingestion could honour.

• The legacy names stay reserved. defaultknowledge and sharedknowledge are rejected as knowledge database names, and default_rag/shared_rag remain in IngestorType and in IngestorEntity.reserved_ids() even though no code reads them any more. Without this, a new database could be created on the name of a frozen corpus and ingested on top of it by a pipeline that does not own it.

  Amended 2026-09-07 (#1835). Reserved originally meant one set, applied to creation and to every read and delete, which made the two legacy databases return 403 on listing documents, opening one, and deleting one — for every user including the superuser, whatever AIHUB_SHOW_LEGACY_KNOWLEDGE said. That is not what reserving a name is for. The two policies are now separate: the names are closed to creation always, while reads, document deletion and namespace deletion are governed by the ordinary per-resource rules, exactly as before the split. The legacy names re-enter the read guard only when the deployment hides legacy knowledge, so that hidden means unreadable and not merely unlisted. Deleting a legacy database as a whole stays refused, in the service, with a message naming the cause.

Consequences

Positive

• One store-construction surface and one naming scheme; the routed_ prefix no longer implies a variant that exists.
• Compose and workspace cannot drift: both derive legacy presence from a single image tag.
• Frozen is expressed structurally — no build context, no floating tag — rather than by convention.
• The playground moves onto the real pipeline, so the dev example and the deployed article are the same code.

Trade-offs

• Legacy bugs can no longer be fixed. A published image cannot receive a patch, and main no longer contains the code to build one. Shipping a legacy fix means branching from the v0.320.1 tag and dispatching build-pipelines.yml against that branch, where compose-config.yml still carries build: localbuild for both images. Branch from v0.320.1, never from the older v0.320.0 release: v0.320.1 is what gave the legacy pipelines a teardown sensor, matching the behaviour the new ingestion pipeline already had, so building from v0.320.0 would reinstate a corpus that can be added to but never pruned. The escape hatch stays open by accident of git history rather than by policy: nothing keeps the old branch buildable as its dependencies age, and the rebuilt image is not digest-identical to the original (python:3.13-slim floats and the apt layer is unpinned).
• A hotfix on release/0.320 must not reuse the v0.320.1 version. That branch still builds the legacy images but never received the teardown sensor, so a build from it would overwrite the frozen images with that same regression. promote.yml derives the next patch from the highest v0.320.* tag, so the v0.320.1 tag keeps a hotfix on that line landing on v0.320.2.
• Nothing removes a torn-down legacy namespace permanently while the seeder runs. AIHUB_CREATE_DEFAULT_BUCKETS (default true) re-creates the two configured bucket and namespace rows on every API start, and init-buckets.sh re-creates the S3 buckets from the same variable. Deleting defaultnamespace itself therefore purges its contents but the empty row returns on the next restart. A deployment that wants it gone sets that variable to false.
• Downstream users of default_definitions break on upgrade with no deprecation window.
• Two dead names are reserved forever, in IngestorType and in the controller's creation-reserved set, long after anything reads them.
• AIHUB_SHOW_LEGACY_KNOWLEDGE gates read access, not just listing. Any deployment still running the frozen images must set it to true; the shipped default is false, and with it off the two databases are neither listed nor browsable. It is read once when the controller is built, so changing it needs an API restart.

Related Decisions

• 2026_06_18_rag_pipeline_route_per_run.md — the configurable pipeline that supersedes these (premise)
• 2026_07_18_self_service_knowledge_deletion.md — legacy databases are undeletable, which is why their names must be reserved rather than freed