log: Added support to include date-time in log filename

[HikaruHokkyokusei]

Added a flag log-file-time-format that allows specifying a date-time format that is used to replace the :dt: wildcard in the filename of the log file.

What is the purpose of this change?

This change allows users to specify a Date-time Format string for a new flag log-file-time-format. A datetime string is generated using this format that replaces the wildcard :dt: in the log filename. This change is required for a specific scenario: -

By default, rclone always appends (not overwrite) logs to a file. This is the same with cases when a Windows Service is created to allow rclone to run in the background while mounting. Based on the logging level, these logs can be huge and one might not prefer them getting appended to the same file every time the mount command is used (say after the device is restarted). Making dynamic filename (in the context of Windows Service) is a tricky challenge, as the %....% variable templates are not replaced with the actual values.

This change allows users to now use dynamic filenames based on the date and time of each session of the rclone.

Was the change discussed in an issue or in the forum before?

While the change has not been discussed in any issues, a couple of users have talked in the context of similar requirements in the forums.

https://forum.rclone.org/t/dynamically-create-log-file-with-date-in-the-name/27603
https://forum.rclone.org/t/rclone-win2019-scheduled-tasks-date-in-logfile/33232

Checklist

• I have read the contribution guidelines.
• I have added tests for all changes in this PR if appropriate.
• I have added documentation for the changes if appropriate.
• All commit messages are in house style.
• I'm done, this Pull Request is ready for review :-)

[ncw]

This is a nice idea.

A couple of thoughts

1. Do we need to invent our own time format when we can just link to the Go docs? Admittedly the Go docs aren't terribly clear, but there are lots of examples.
2. I bet that someone made a nice tested library implementing strftime formatting - eg this one https://github.com/ncruces/go-strftime if we really want to use it.
3. This could be used in other places, eg
   • --suffix (The backup-dir/suffix one has been requested many times.)
   • --backup-dir
   • --log-file
   • The output file options for rclone sync and rclone check (eg --combined, --differ, --match etc)
4. We should maybe be offering the user the choice of substitution pattern, so make the :dt: configurable

This might lead to a UI which looked like --file-time-format ID=FORMAT so you might say --file-time-format @@=%Y-%m-%d --suffix .@@.backup

What do you think @HikaruHokkyokusei ? @nielash do you have any thoughts?

[nielash]

1. Do we need to invent our own time format when we can just link to the Go docs?

I did something just like this recently (actually twice!) Maybe this should be factored out as a lib package?

rclone/fs/operations/operations.go

Lines 2106 to 2156 in 2bebbfa

	// AddModTime adds file's Mod Time to output
	func (l *ListFormat) AddModTime(timeFormat string) {
	switch timeFormat {
	case "":
	timeFormat = "2006-01-02 15:04:05"
	case "Layout":
	timeFormat = time.Layout
	case "ANSIC":
	timeFormat = time.ANSIC
	case "UnixDate":
	timeFormat = time.UnixDate
	case "RubyDate":
	timeFormat = time.RubyDate
	case "RFC822":
	timeFormat = time.RFC822
	case "RFC822Z":
	timeFormat = time.RFC822Z
	case "RFC850":
	timeFormat = time.RFC850
	case "RFC1123":
	timeFormat = time.RFC1123
	case "RFC1123Z":
	timeFormat = time.RFC1123Z
	case "RFC3339":
	timeFormat = time.RFC3339
	case "RFC3339Nano":
	timeFormat = time.RFC3339Nano
	case "Kitchen":
	timeFormat = time.Kitchen
	case "Stamp":
	timeFormat = time.Stamp
	case "StampMilli":
	timeFormat = time.StampMilli
	case "StampMicro":
	timeFormat = time.StampMicro
	case "StampNano":
	timeFormat = time.StampNano
	case "DateTime":
	// timeFormat = time.DateTime // missing in go1.19
	timeFormat = "2006-01-02 15:04:05"
	case "DateOnly":
	// timeFormat = time.DateOnly // missing in go1.19
	timeFormat = "2006-01-02"
	case "TimeOnly":
	// timeFormat = time.TimeOnly // missing in go1.19
	timeFormat = "15:04:05"
	}
	l.AppendOutput(func(entry *ListJSONItem) string {
	return entry.ModTime.When.Local().Format(timeFormat)
	})
	}

(docs)

rclone/cmd/bisync/bilib/names.go

Lines 107 to 165 in 2bebbfa

	// TimeFormat converts a user-supplied string to a Go time constant, if possible
	func TimeFormat(timeFormat string) string {
	switch timeFormat {
	case "Layout":
	timeFormat = time.Layout
	case "ANSIC":
	timeFormat = time.ANSIC
	case "UnixDate":
	timeFormat = time.UnixDate
	case "RubyDate":
	timeFormat = time.RubyDate
	case "RFC822":
	timeFormat = time.RFC822
	case "RFC822Z":
	timeFormat = time.RFC822Z
	case "RFC850":
	timeFormat = time.RFC850
	case "RFC1123":
	timeFormat = time.RFC1123
	case "RFC1123Z":
	timeFormat = time.RFC1123Z
	case "RFC3339":
	timeFormat = time.RFC3339
	case "RFC3339Nano":
	timeFormat = time.RFC3339Nano
	case "Kitchen":
	timeFormat = time.Kitchen
	case "Stamp":
	timeFormat = time.Stamp
	case "StampMilli":
	timeFormat = time.StampMilli
	case "StampMicro":
	timeFormat = time.StampMicro
	case "StampNano":
	timeFormat = time.StampNano
	case "DateTime":
	// timeFormat = time.DateTime // missing in go1.19
	timeFormat = "2006-01-02 15:04:05"
	case "DateOnly":
	// timeFormat = time.DateOnly // missing in go1.19
	timeFormat = "2006-01-02"
	case "TimeOnly":
	// timeFormat = time.TimeOnly // missing in go1.19
	timeFormat = "15:04:05"
	case "MacFriendlyTime", "macfriendlytime", "mac":
	timeFormat = "2006-01-02 0304PM" // not actually a Go constant -- but useful as macOS filenames can't have colons
	}
	return timeFormat
	}
	// AppyTimeGlobs converts "myfile-{DateOnly}.txt" to "myfile-2006-01-02.txt"
	func AppyTimeGlobs(s string, t time.Time) string {
	hasGlobs, substring := ParseGlobs(s)
	if !hasGlobs {
	return s
	}
	timeString := t.Local().Format(TimeFormat(TrimBrackets(substring)))
	return strings.ReplaceAll(s, substring, timeString)
	}

(docs)

[HikaruHokkyokusei]

@nielash Thanks for pointing out the specific code where you have implemented this.

@ncw , Here is why I believe we cannot reuse the code provided by @nielash : -
Since we will be using this time in the name of the file, where / and : might not be allowed on some platforms, our available options will be severely restricted after eliminating invalid options. One possibility is to use the same technique that rclone uses to replace /, :, etc. in the filenames with other similar Unicode characters like ／ and ∶ when it is syncing, but will that be worth it?

Also, thanks for pointing out go-strftime repo. I had come across it before, but didn't know if we were allowed to use external libs. I believe this is the best approach so I will replace my implementation with go-strftime lib. What do you guys think?

---

I will take a look at other places that you have mentioned in point number 3 and let you know @ncw .

---

Yep, the idea of a dynamic wildcard (instead of just using :dt:) makes sense. I'll make the appropriate changes for that too. I will make it such that the flag format will look like this --file-time-format WILDCARD~#~FORMAT. The argument value will be split at ~#~ and will provide us with both, the wildcard and the format.

[HikaruHokkyokusei]

So with this commit, I have taken care of the custom wildcard and replaced the formatting methodology with the go-strftime library.

@ncw , I looked into --suffix and --backup-dir. I don't want to make changes to them without having a deep understanding of what is being done there.

I do not know the frequency of how many times these values are referred to. The logging file was created once, so that was straightforward.

One idea I had was to set the value for ci.Suffix and ci.BackupDir in the SetFlags function here which is called at initialization time here.

In this SetFlags function, I plan to call log.Opt.FilenameTimeFormat and log.Opt.FilenameTimeWildcard defined here.

This way, one specific value will be set for --suffix and --backup-dir. Is this enough? I see in the code that operations like opt.OrigBackupDir = ci.BackupDir are performed here. So, the question arises whether I should set this value just once when the rclone is being initialized, or to recalculate this suffix or backup-dir wherever they are used, which depends on the frequency of them being called and their use case.

[nielash]

I think your solution is fine, but just to clarify one thing:

Since we will be using this time in the name of the file, where / and : might not be allowed on some platforms, our available options will be severely restricted after eliminating invalid options.

This issue is specifically addressed in bisync's --conflict-suffix by inventing a new constant without colons or slashes:

rclone/docs/content/bisync.md

Lines 628 to 635 in f62e7b5

	All of the formats described [here](https://pkg.go.dev/time#pkg-constants) and
	[here](https://pkg.go.dev/time#example-Time.Format) are supported, but take
	care to ensure that your chosen format does not use any characters that are
	illegal on your remotes (for example, macOS does not allow colons in
	filenames, and slashes are also best avoided as they are often interpreted as
	directory separators.) To address this particular issue, an additional
	`{MacFriendlyTime}` (or just `{mac}`) option is supported, which results in
	`2006-01-02 0304PM`.

and also allowing the user to specify their own custom format:

rclone/cmd/bisync/bilib/names.go

Line 154 in 2bebbfa

return timeFormat

That said, I agree that %Y-%m-%d semantics might be more intuitive than 2006-01-02 to casual users who aren't Go developers! So perhaps go-strftime is a better choice for this reason.

[HikaruHokkyokusei]

@ncw , would be great if you could provide some insight about how to move forward with --suffix and --backup-dir, otherwise I believe this PR is good to be merged.

[HikaruHokkyokusei]

@nielash @ncw , I think this PR can be merged unless we are waiting for something else.

[ncw]

Sorry for the delay merging @HikaruHokkyokusei

I think the code and implementation looks great but I'd like to discuss exactly what this looks like as these things are very difficult to change once released!

We have at the moment --file-time-format WILDCARD~#~FORMAT so the user might use something like --file-time-format @~#~%Y-%m-d which means for @ substitute the time 2001-02-03 in --log-file rclone.@.log

I'm thinking ahead a bit here, but I'd like to re-use this syntax in --backup-dir and --suffix which would relieve a small but significant pain point in rclone. This syntax could also be used in remote strings on the command line which is something people often want to do - copy to a dated folder with `rclone copy .... src: dst:@Date@/

So up for discussion

1. The name of the flag --file-time-format - maybe it should be more generic --time-format if we are thinking of re-using it with --suffix and --backup-dir, thought it does apply to files always.
2. Should it have a default? Say --file-time-format @date@~#~%Y-%m-d. I think probably not but I don't know.
3. I don't like the ~#~ separator. What is wrong with using =? That would mean users couldn't use = as part of the wildcard, but there are plenty of other characters which can be used and that makes it very similar to other rclone flags.
4. We could use @nielash standard forms here also if we had --time-format @@=RFC3339 where the contents after the = was exactly one of those standard forms.

Any thoughts welcome (@nielash and @albertony also)