New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Database reset with encrypted folders causes a full resync #9493
Comments
I was with you up to
how did that happen? AFAIK the file names should be the same after the resync. |
Well, I'm not joking.
Size on disk: 1.8 GB. Before DB reset it was 920 MB. After the DB reset, syncthing made a full resync. After clicking on "Revert local changes": After pause and then unpause the folder, "Locally changed items" disappears. Edit: |
Probably conflict copies, now that I think about it, which I think is also expected. |
Possible, I'll check that. But why are the files conflicting? No key or file was changed. Additionally, the bug with not resetting the counter for locally changed files does not reset after clicking on revert. Just the file size is changed to 0. After pause and unpause, this is fixed. Possible related to PR #9319?! Edit: Should be file conflicts a thing with "Receive encrypted" folders at all? |
I understand now that encrypted folders are never hashed (see https://github.com/syncthing/syncthing/blob/main/lib/model/folder.go#L664). So syncthing is always assuming that the encrypted side is outdated when the index DB is reset/broken/corrupted. Index DB corruption happened to me a few times already. Downsides:
|
Resetting the database is not a normal operational thing, I would rather focus on the cause of the corruption you're seeing. I've been running this thing for a decade now on uncounted boxes and haven't yet got a corrupted database. |
It happened more than once after one VPS provider did some overselling and the HDD performance dropped very much during peek hours. Of course it should not be corrupted, but things can break always. Also a HDD can be corrupted, sectors can break, RAM can malfunction or bits can flip. Without deep verification of the data stored on the disk, the integrity is not guaranteed. Sadly for me the encrypted folders sound like a half production ready solution. |
Well, not just for you as the docs regarding this feature are very clear about its state;
I'm not entirely sure what's needed to get it out of that state, but I can imagine that the lack of gracefully handling such corner cases is a part of it. Together with some pending bugs. |
Thanks for pointing out the documentation. As I understand it right, the current protocol makes it impossible to verify single blocks.
The only way I see is to implement this is by storing the hash of the encrypted data on the trusted side, alongside the hash of the plain data. The cause will be a bigger index db, more memory usage (?), and more CPU usage during the hashing process. Is such a change wanted to syncthing? Or are there any other ideas or plans? |
I don't think it's desired in general, no. |
So what other plans are there to get the encryption feature out of the beta status? |
I don't see that this is related to that in any way? I think the beta-ness is about the amount of testing, and possibly a couple of known bugs with the storage format that should be fixed. Your issue, paraphrasing, is that you used a nuclear-level debugging option (that we arguably shouldn't even have) because of potential hardware issues, and the resulting user experience wasn't smooth, which isn't surprising to me. |
It's not only not smooth, I believe the missing and unplanned untrusted-side verification is a design flaw and makes syncthing an bad choice for some use cases. As said before, corruption of storage media can and does happen, as well as db corruption. I rather switch to tools like restic to store my data on an untrusted server since this tool provides industry standards for data safety. |
What happened?
syncthing --reset-database
Syncthing will detect all already synced data as "Locally changed" and will do a full resync of all data.
After this, storage consumption is 2x. This can be fixed with "Revert local changes".
Aside from this it happens that there are still "Locally changed items", even after pressing on "Revert local changes". Pause and unpause the folder sometimes helps.
Syncthing version
v1.27.4
Platform & operating system
Linux arm64
The text was updated successfully, but these errors were encountered: