Stale File Handles : Auto-Healing Docker Mounts after Network Storage Drops

In the previous parts of this series, I detailed how I migrated a 10TB external media drive to OpenMediaVault (OMV) and decoupled my container configurations (appdata) from the primary host VM (Docker Host) by mounting them over NFS. That transition moved me closer to a 100% stateless and disposable Docker VM.

But this decoupled architecture introduced a new operational headache: the network storage drop.

Whenever the OMV storage VM rebooted for maintenance or the internal virtual network hiccuped, running containers lost connection to their directories. Even when the host host-mount recovered automatically, containers remained locked in a broken state in NZBGet, throwing a frustrating error: ESTALE (errno=116): Stale file handle.

Here's how I automated the detection and auto-healing of stale NFS mount points inside Docker container namespaces.

The Root Cause: Inode Mismatch

When a Docker container starts with a bind mount (e.g., mapping /mnt/Element10tb/data on the host to /data in the container), the Linux kernel binds the container's mount namespace to the specific filesystem inode identifier on the host.

If the NFS share drops and then reconnects:

1. The host re-establishes the connection and mounts the share, creating a new filesystem inode.
2. The running container is unaware of this change and continues pointing to the old, dead inode.
3. The application inside the container (e.g., Sonarr, Radarr, or NZBGet) throws a Stale file handle error on any disk access, displaying "Space Unknown" or failing file writes.
4. The storage remains completely broken for the application until the container namespace is restarted, forcing it to bind to the new host inode.

┌─────────────────┐             ┌─────────────────┐             ┌─────────────────┐
│     OMV NAS     │             │   Docker Host   │             │ Docker Container│
│  (NFS Server)   │             │   (VM Parent)   │             │   (Namespace)   │
└────────┬────────┘             └────────┬────────┘             └────────┬────────┘
         │                               │                               │
         │ 1. Mounts Share (Inode A)     │                               │
         ├──────────────────────────────>│                               │
         │                               │ 2. Binds /data to Inode A     │
         │                               ├──────────────────────────────>│
         │                               │                               │
    [ NAS Drops & Reboots ]              │                               │
         │                               │                               │
         │ 3. Remounts Share (Inode B)   │                               │
         ├──────────────────────────────>│                               │
         │                               │                               │
         │                               │ 4. Reads /data (Points to A)  │
         │                               │<──────────────────────────────┤
         │                               │                               │
         │                               │ 5. Returns Error (Inode dead) │
         │                               ├──────────────────────────────>│
         │                               │    [ ESTALE / Stale Handle ]  │

Designing the Auto-Healer

To solve this without manual intervention, I wrote a lightweight recovery script that runs periodically on the host. The script performs a two-tier verification:

1. Host-Level Check: Verifies if the host mount directory /mnt/Element10tb is accessible. If not, it force-unmounts, remounts, and restarts all dependent containers.
2. Container-Namespace Check: If the host is healthy, it executes a filesystem query inside each running container's namespace. If it detects a Stale file handle error, it automatically restarts that specific container.

The Auto-Healing Script

Here is the recovery script, /home/deploy/docker/scripts/monitor-nfs-mounts.sh:

#!/bin/bash
# ==============================================================================
# Monitor NFS Mounts and Restart Dependent Docker Containers on Stale Handles
# ==============================================================================

# Configuration
MOUNT_PATH="/mnt/Element10tb"
DEPENDENT_CONTAINERS=(
    "qbittorrent" 
    "sonarr" 
    "radarr" 
    "nzbget" 
    "prowlarr" 
    "lidarr" 
    "slskd" 
    "filebrowser_quantum" 
    "tubearchivist" 
    "iplayarr"
)
LOG_FILE="/home/deploy/docker/scripts/monitor-nfs-mounts.log"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# 1. Check if the host mount path is healthy
if ! ls "$MOUNT_PATH/data" >/dev/null 2>&1; then
    log "WARNING: Host mountpoint $MOUNT_PATH is stale or empty."
    log "Attempting to unmount $MOUNT_PATH..."
    sudo umount -f -l "$MOUNT_PATH" >> "$LOG_FILE" 2>&1
    
    log "Attempting to remount $MOUNT_PATH..."
    sudo mount "$MOUNT_PATH" >> "$LOG_FILE" 2>&1
    
    if ls "$MOUNT_PATH/data" >/dev/null 2>&1; then
        log "SUCCESS: Remounted. Restarting dependent containers..."
        for container in "${DEPENDENT_CONTAINERS[@]}"; do
            docker restart "$container" >> "$LOG_FILE" 2>&1
        done
    else
        log "CRITICAL: Remount failed. Manual intervention required!"
    fi
else
    # 2. Host is healthy; check inside each container's namespace
    for container in "${DEPENDENT_CONTAINERS[@]}"; do
        if docker ps --format '{{.Names}}' | grep -q "^$container$"; then
            STALE=false
            
            # Query the directory inside the container
            if docker exec "$container" ls /data >/dev/null 2>&1; then
                :
            elif [ $? -eq 1 ] || docker exec "$container" ls /data 2>&1 | grep -q "Stale file handle"; then
                STALE=true
            fi
            
            if [ "$STALE" = true ]; then
                log "WARNING: Container $container has a stale mount handle inside. Restarting..."
                docker restart "$container" >> "$LOG_FILE" 2>&1
            fi
        fi
    done
fi

Scheduling the Daemon

To make the check hands-off, I scheduled the script to execute every 5 minutes under the host's crontab:

*/5 * * * * /home/deploy/docker/scripts/monitor-nfs-mounts.sh

Lessons Learned & Best Practices

1. Do Not Store Appdata Databases on NFS: Active databases (SQLite, PostgreSQL, Elasticsearch) require robust file locks. NFS has poor locking support, meaning network stutters will cause database corruption. Keep appdata/config on local SSD storage, and use NFS strictly for bulk storage (media, downloads).
2. Audit All Dependent Mounts: Ensure every container that bind-mounts a network path is included in your auto-healing script (including helper tools like Filebrowser and Soulseek).
3. Use Lazy Unmounts (umount -l): If an NFS path goes stale, a standard umount will hang indefinitely. A lazy unmount detaches the filesystem immediately, letting you run a clean remount while the kernel cleans up the dead connections in the background.

Don't forget to explore the rest of our website as we build out more content. Stay tuned for more tutorials, tips, and tricks to help you make tech work for you.

If you want to stay up-to-date with regular updates, make sure to subscribe to our free mailing list.