The historical dat file has not been truly deleted, and the disk is full

[lisabiya]

Describe the bug
I have investigated bagerDB, boltDB and nusDB before, and I think that the bucket design of nutsDB has Redis like data structures and operators that meet my product requirements
At present, I am trying to use Nutsdb in a production environment, and it has been running very well. However, a few days later, I received a server alarm saying that the hard drive is full.
lsof |grep deleted After querying, it was found that many files were not actually deleted, thus occupying a large amount of space

Give sample code if you can.
init

func (nd *NutsDb) SetupDb(log *log.Logger) {
	var options = nutsdb.DefaultOptions
	options.MergeInterval = time.Minute * 30
	db, err := nutsdb.Open(
		options,
		nutsdb.WithDir(nd.Dir),
	)
	if err != nil {
		log.Println("NutsDb*SetupDb err", err.Error())
	} else {
		log.Println("***********************NutsDb*SetupDb*******************")
	}
	nd.db = db
}

Expected behavior
Expected scheduled deletion of files

What actually happens
Disk occupancy is extremely high

Screenshots

please complete the following information :

• OS: [Ubuntu 20]
• NutsDB Version [0.13.1]

Additional context
Looking forward to your reply

[elliotchenzichang]

Hi @lisabiya I am one of the main maintainers for this project. Did not fully understand what file should be deleted in your use case. Could you help explain more about those files?

[bigboss2063]

Hello @lisabiya, Merge will only remove invalid data and then rewrite valid data to the disk instead of regularly deleting all files. And we recommend updating nutsdb to the latest version. We have fixed some bugs and made some optimizations.

[lisabiya]

Hello @lisabiya, Merge will only remove invalid data and then rewrite valid data to the disk instead of regularly deleting all files. And we recommend updating nutsdb to the latest version. We have fixed some bugs and made some optimizations.

Thank you for your reminder. I have now updated the version to 0.14.1 and will continue to observe the data. However, it may take several days for the data to accumulate before it appears. I will provide feedback on the structure. Thank you again for your help

[bigboss2063]

If you want to delete all files containing key-value pairs that have not been deleted, it may be more appropriate to set an expiration deletion time.😁 @lisabiya

[lisabiya]

Hi @lisabiya I am one of the main maintainers for this project. Did not fully understand what file should be deleted in your use case. Could you help explain more about those files?
Thank you for your reply😀
Hello, I am currently using it to record and synchronize some temporary data. Simply put, I am doing a temporary data cache for user information, which will cache for 30 minutes and then expire. The daily data read and write cache is around a million level. The problem I am currently facing is that after running for about a week, the program will find that disk storage is starting to accumulate and there is no downward trend. By querying * * lsof | grep deleted * *, Many deleted files can be found, but the program is still using them, resulting in the actual disk not being released. The disk will only return to normal after I restart the program
The following is the code for storing and updating cache expiration times that I mainly use

func (nd *NutsDb) SetData(bucket string, key string, val interface{}, ttl uint32) (err error) {
	return nd.db.Update(
		func(tx *nutsdb.Tx) error {
			marshal, err := nd._marshal(val)
			if err != nil {
				return err
			}
			err = tx.Put(bucket, []byte(key), marshal, ttl)
			if err != nil {
				return err
			}
			return nil
		})
}

func (nd *NutsDb) SetOrUpdateExpiredTime(bucket string, key string, val interface{}, ttl uint32) (err error) {
	return nd.db.Update(
		func(tx *nutsdb.Tx) error {
			get, err := tx.Get(bucket, []byte(key))
			if err != nil { 
				marshal, err := nd._marshal(val)
				if err != nil {
					return err
				}
				err = tx.Put(bucket, []byte(key), marshal, ttl)
				if err != nil {
					return err
				}
			} else { 
				err = tx.Put(bucket, []byte(key), get.Value, ttl)
				if err != nil {
					return err
				}
			}
			return nil
		})
}

Disk monitoring

[lisabiya]

Unfortunately, when I replaced it with 0.14.1, the problem still exists

What I have identified is the most frequently used

 tx.Put(bucket, []byte(key), marshal, ttl)

For configuring caching, I have a question. If the same key is used for overwriting storage, will the old data be released?
I have written a use case to verify this issue. Currently, it seems that there is a situation where the occupation has not been released, but I am not sure if such a problem really exists

Here is an example

package test

import (
	"fmt"
	"github.com/nutsdb/nutsdb"
	"log"
	"testing"
	"time"
)

var nutDbs *nutsdb.DB

const second uint32 = 1
const minute uint32 = second * 60

func TestNutsDb_SetOrUpdateExpiredTime(t *testing.T) {
	var options = nutsdb.DefaultOptions
	options.MergeInterval = time.Minute * 30
	var err error
	nutDbs, err = nutsdb.Open(
		options,
		nutsdb.WithDir("proxyDb"),
	)
	if err != nil {
		panic(err)
	}

	var ticker = time.NewTicker(time.Second * 10)
	for {
		select {
		case <-ticker.C:
			for i := 0; i < 10000; i++ {
				err := SetOrUpdateExpiredTime("bucket", fmt.Sprintf("key%d", i), "valuevaluevaluevaluevaluevaluevaluevaluevaluevaluevaluevaluevalue", 30*second)
				if err != nil {
					log.Println(err.Error())
				}
			}
			log.Println("task finish")
		}
	}
}

func SetOrUpdateExpiredTime(bucket string, key string, val string, ttl uint32) (err error) {
	return nutDbs.Update(
		func(tx *nutsdb.Tx) error {
			get, err := tx.Get(bucket, []byte(key))
			if err != nil { //Not existing, update
				err = tx.Put(bucket, []byte(key), []byte(val), ttl)
				if err != nil {
					return err
				}
			} else { //Update existing data
				err = tx.Put(bucket, []byte(key), get.Value, ttl)
				if err != nil {
					return err
				}
			}
			return nil
		})
}

[elliotchenzichang]

I think we may meet this issue:
https://access.redhat.com/solutions/2316

And it's saying that:
On Linux or Unix systems, deleting a file via rm or through a file manager application will unlink the file from the file system's directory structure; however, if the file is still open (in use by a running process) it will still be accessible to this process and will continue to occupy space on disk. Therefore such processes may need to be restarted before that file's space will be cleared up on the filesystem.

[bigboss2063]

Thanks for the detailed description.

When writing a new value with the same Key, the old data on the disk will not be released immediately, and will not be released until Merge is triggered.

I checked your description again. The reason may be that after deleting the file, the file descriptor is still held by the nutsdb instance and the disk occupation is not released, so it is released after restarting.

[bigboss2063]

A fix will be made after we verify @lisabiya