Image

How I Saved My Immich Installation from a Broken PostgreSQL Upgrade on TrueNAS

#immich#postgres#truenas#self-hosting#warp

The Problem

I run Immich on my TrueNAS server — it’s my self-hosted alternative to Google Photos. I’ve been slowly moving over to Immich, and so far I have about 5,000 photos over there. I’m migrating the rest incrementally from Google Photos and my Lightroom collection. It’s a process, but Immich has been solid — until it wasn’t.

One day, I noticed Immich wasn’t starting. The TrueNAS UI was showing a cryptic error:

[EFAULT] Failed ‘up’ action for ‘immich’ app. Please check /var/log/app_lifecycle.log for more details.

Not exactly helpful on its own. I was on chart version 1.13.6, and the UI was prompting me to upgrade to 1.14.5. I tried upgrading through the TrueNAS UI, but even the upgrade would fail. The reason, as I would later discover, was that iXSystems had remotely removed PostgreSQL 15 completely from the app chart. So when TrueNAS tried to start the container, it would hand it the existing PostgreSQL 15 data volume, but the container only knew about PostgreSQL 18 — and it would crash.

Digging Into the Logs

I had tried analyzing the log files myself, but the log was very poorly formatted — everything was crammed into long, unstructured lines with escaped newlines, mixed with pull progress bars and lifecycle events from every app on the system. It was a wall of noise. I knew that Warp’s AI agent is built for exactly this kind of thing — making sense of messy system output — so I immediately switched to Warp.

Using the AI agent, I asked it to analyze /var/log/app_lifecycle.log. Within seconds, it identified the root cause buried deep in the logs:

service "pgvecto_upgrade" didn't complete successfully: exit 1

Drilling deeper into the container logs revealed the real culprit:

ERROR: Old PostgreSQL [15] binaries not found at [/usr/lib/postgresql/15/bin]

The pgvecto_upgrade container was trying to upgrade my database from PostgreSQL 15 to 18, but it couldn’t find the PG15 binaries needed to perform the migration.

Understanding the Gap

This is where Warp’s agent really shined. It dug into the TrueNAS catalog’s git history and pieced together the full timeline:

  • Chart version 1.13.6 (my version) included references to both postgres:15-vectorchord and postgres:18-vectorchord images
  • Chart version 1.14.0 (commit ec0b5a0235) explicitly removed the PG15 image reference with the message: “apps: remove pg15 from some apps that was missed”
  • Chart versions 1.14.1–1.14.5 all lacked PG15 support

The upgrade container image told the same story. The current ixsystems/postgres-upgrade:1.2.1 only shipped with PG 16, 17, and 18 binaries. PG15 was gone. But an older version, ixsystems/postgres-upgrade:1.1.0, still had PG 15, 16, 17, and 18.

There was likely a brief window — maybe days — where the catalog had the right combination of images to smoothly upgrade from PG15 to PG18. I missed that window. And now I was stuck.

The Upgrade Path Problem

This highlights a broader issue with how the TrueNAS app catalog handles major PostgreSQL version upgrades. The template only offered two options: PostgreSQL 15 and PostgreSQL 18. There was no iterative upgrade path — no stepping through PG16 or PG17. If you missed the narrow window where the upgrade tooling supported your current version, you were out of luck.

A more gradual approach — supporting upgrades across adjacent major versions and keeping older binaries available for longer — would have prevented this entirely. Database major version upgrades are one of the most delicate operations in any stack, and they deserve a more robust migration strategy.

The Fix: A Manual Migration

Since the automated upgrade path was broken, I had to perform a manual migration. Here’s what I did with Warp’s agent guiding me through every step:

1. Attempted the Quick Fix First

I tried editing the TrueNAS catalog files to swap the upgrade image from 1.2.1 to 1.1.0 (which had PG15 binaries). But TrueNAS re-renders its compose files from cached templates every time you save from the UI, overwriting my changes. I learned that TrueNAS stores rendered configs in /mnt/.ix-apps/app_configs/, separate from the catalog source in /mnt/.ix-apps/truenas_catalog/ix-dev/.

2. Ran pg_upgrade — Almost Worked

After fixing the image in the right location and running the upgrade container manually, pg_upgrade got further but hit another wall: the generic upgrade image didn’t have Immich’s custom PostgreSQL extensions (vchord and vector).

3. The pg_dump/pg_restore Approach

This is what ultimately worked:

  1. Started a temporary PG15 container using the Immich-specific postgres image (ghcr.io/immich-app/postgres:15-vectorchord0.5.3) with the old data directory
  2. Dumped the entire database — 157MB of data
  3. Moved the old PG15 data out of the postgres data directory
  4. Started a fresh PG18 container using the Immich PG18 image (ghcr.io/immich-app/postgres:18-vectorchord0.5.3)
  5. Restored the dump into the fresh PG18 instance
  6. Started Immich through TrueNAS

4. One More Hurdle: Password Mismatch

After the migration, Immich’s web UI wouldn’t load. The server logs showed password authentication failed for user "immich". When I used midclt call app.update to register the PG18 config with TrueNAS, it re-rendered the compose file with a new auto-generated password. But my manually-initialized PG18 database still had the old password. A quick ALTER USER fixed it, and Immich came to life — fully running on PostgreSQL 18.2 with all 5,590 assets intact.

How Warp Made This Possible

I want to highlight how much Warp simplified this entire process. Without Warp’s AI agent:

  • I would have spent hours manually reading through massive log files trying to find the relevant error
  • I wouldn’t have thought to dig through the git history of the TrueNAS catalog to understand the version timeline
  • I might not have discovered that ixsystems/postgres-upgrade:1.1.0 had the PG15 binaries I needed
  • The multi-step manual migration — involving temporary containers, dumps, restores, and config adjustments — would have been far more error-prone

Instead, I had an AI agent that could inspect container images, read configs, trace through the TrueNAS middleware API, and guide me through each step. It felt like pair-programming with someone who had deep knowledge of Docker, PostgreSQL, and TrueNAS internals.

Most importantly, I didn’t have to delete my app and restore from backups. The data was preserved throughout the entire process. That’s the difference between a surgical fix and a sledgehammer approach.

Lessons Learned

  1. Stay on top of updates, even if it’s just a few days. Software moves incredibly fast. The window for a smooth PG15→PG18 migration was brief, and missing it meant a manual intervention. What seems like a harmless delay of a few days can close an upgrade path entirely.

  2. Major database version upgrades deserve better tooling. Jumping from PG15 straight to PG18 with no intermediate steps is risky. The upgrade images should support broader version ranges, and the migration path should be more gradual and forgiving.

  3. The right tools make impossible problems solvable. What could have been a data-loss scenario or a multi-hour manual recovery turned into a guided, systematic debugging session. Having an AI agent that understands the full stack — from log files to Docker images to TrueNAS internals — is a game-changer for self-hosted infrastructure.

  4. Always have backups, but know that you might not need them. I had backups ready, but the targeted approach meant I never had to reach for them. Understanding the system deeply enough to fix it in place is always preferable to a full restore.

This post documents my experience on March 31, 2026, upgrading Immich’s PostgreSQL database from version 15 to 18 on TrueNAS SCALE, with the help of Warp’s AI terminal agent.