After host replacement, seaweedfs instance on the new host is not synced with other instances

[Meenakshi-Sachdeva]

Describe the bug
Running seaweedfs on 3 host system(HA) 0, 1, 2, with one master, volume and filer on each instance. The cluster runs fine. Then we replace host 2 with a new host and form a new cluster. When running cluster.ps or cluster.check, the cluster is fine. But when checking "s3cmd ls" in new host, no bucket is coming whereas host 0 and 1 show all the data.

(Cmd) enter_container seaweedfs_0
root@cd67a440b3d6:/# s3cmd ls
2024-03-12 00:07  s3://dr-backup
2024-03-11 23:06  s3://fcw-artifacts
root@cd67a440b3d6:/# exit
exit
(Cmd) enter_container seaweedfs_1
root@65c7c085de22:/# s3cmd ls
2024-03-12 00:07  s3://dr-backup
2024-03-11 23:06  s3://fcw-artifacts
root@65c7c085de22:/# exit
exit
(Cmd) enter_container seaweedfs_2
root@2b245735f416:/# s3cmd ls
root@2b245735f416:/# 

System Setup

• List the command line to start "weed master", "weed volume", "weed filer", "weed s3", "weed mount".
• OS version Ubuntu 20.04.6 LTS
• output of weed version 3.63
• if using filer, show the content of filer.toml not using filer.toml

Expected behavior
s3cmd ls should show same output in all container

Additional context
As a workaround, I have saved metadata from container 0(fs.meta.save -o filer.meta) and loaded it to container 2 (fs.meta.load filer.meta)

[sgalsaleh]

I hit this issue in Kubernetes as well when using the Helm chart to deploy SeaweedFS and replacing one of the hosts. I used the same workaround as you which is to save the filer metadata from another filer and load it into the new filer instance. Great to see you have a PR with a fix!