Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Node local temporary files/dirs
08-07-2017, 05:07 PM (This post was last modified: 11-16-2017 10:42 AM by zma.)
Post: #1
Node local temporary files/dirs
It is common for programs to have some temporary files like for

logging: http://tab.d-thinker.org/showthread.php?tid=8242

tmp dir: http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=8243&pid=4675

These programs should careful choose the names/paths to avoid conflict while there is no guarantee.

Let's use a common convention so that programs can make use and be sure it is free from conflicts with programs following the convention:

For a program named ${progname} executed by user ${USER}, if it want to use a temporary files/dirs, the common way is to use following ways.

If it is user specific:

/thinker/local/today/users/$USER/${progname}/${file_or_dir_name}

where the ${file_or_dir_name} is chosen by the program.

If it is general ones:

/thinker/local/today/all/$USER/${progname}/${file_or_dir_name}

The programs are recommended to clean these files after they are not used. If files under /thinker/local/today/ are more than 24 hours (by `mtime`) old, they may be deleted automatically or by some other programs. If a program wants to keep the file in /thinker/local/today/ for more than 24 hours, the program should renew its "lease" by updating the file's mtime (such as `touch /thinker/local/today/<path to the file>) before it is 24 hours old.

If there may be multiple program instances running and the programs would like to use unique name for each instance, it can use various ways suitable to the program logic. 2 possible choices:

- $(mktemp -p /thinker/local/today/users/$USER/${progname}/) to get a unique file
- /thinker/local/today/users/$USER/${progname}/${BASHPID} to get a file based on its pid.
Visit this user's website Find all posts by this user
Quote this message in a reply
08-07-2017, 05:10 PM
Post: #2
RE: Node local temporary files/dirs
Ray: please review.
Visit this user's website Find all posts by this user
Quote this message in a reply
08-07-2017, 05:19 PM
Post: #3
RE: Node local temporary files/dirs
(08-07-2017 05:07 PM)zma Wrote:  /thinker/local/$USER/${progname}/${file_or_dir_name}

The program is responsible to clean these files after they are not used.

I think a temp dir under /tmp would be better because some programs may not be able to clear their own tmp files, like filelock.

Under /tmp, the files can be cleared automatically if they are not used for a long time.
Find all posts by this user
Quote this message in a reply
08-07-2017, 05:25 PM
Post: #4
RE: Node local temporary files/dirs
(08-07-2017 05:19 PM)rayluk Wrote:  
(08-07-2017 05:07 PM)zma Wrote:  /thinker/local/$USER/${progname}/${file_or_dir_name}

The program is responsible to clean these files after they are not used.

I think a temp dir under /tmp would be better because some programs may not be able to clear their own tmp files, like filelock.

Under /tmp, the files can be cleared automatically if they are not used for a long time.

Then how shall programs avoid conflicts? /tmp is a system specific dir and many program besides of our ones use it too.

The auto-cleaning feature causes trouble. It cleans files programs don't want them to be cleaned. For files in /tmp that needs to be live longer such as for logs, programs should be configure systemd to avoid cleaning them.

And /tmp is a in-memory fs. All data there consumes memory.

Small files like those used by filelock are fine to be left there if it is small and overwritten next time.
Visit this user's website Find all posts by this user
Quote this message in a reply
08-07-2017, 05:26 PM
Post: #5
RE: Node local temporary files/dirs
(08-07-2017 05:25 PM)zma Wrote:  
(08-07-2017 05:19 PM)rayluk Wrote:  
(08-07-2017 05:07 PM)zma Wrote:  /thinker/local/$USER/${progname}/${file_or_dir_name}

The program is responsible to clean these files after they are not used.

I think a temp dir under /tmp would be better because some programs may not be able to clear their own tmp files, like filelock.

Under /tmp, the files can be cleared automatically if they are not used for a long time.

Then how shall programs avoid conflicts? /tmp is a system specific dir and many program besides of our ones use it too.

The auto-cleaning feature causes trouble. It cleans files programs don't want them to be cleaned. For files in /tmp that needs to be live longer such as for logs, programs should be configure systemd to avoid cleaning them.

And /tmp is a in-memory fs. All data there consumes memory.

Small files like those used by filelock are fine to be left there if it is small and overwritten next time.

Good to me then
Find all posts by this user
Quote this message in a reply
10-24-2017, 11:01 AM (This post was last modified: 10-24-2017 11:07 AM by lingu.)
Post: #6
RE: Node local temporary files/dirs
A program can never guarantee to clean its state, including temp files. But that's fine. We recommend they clean their temp files, and we implement some cleaning utility later to clean old files. The real essential agreements are:
1. where the temp files are. /thinker/local/$USER/${progname} is a bit brutal -- it means no user can store non-temp file in that dir and, then, with a mixture of temp and permanent files, no cleaning utility can work reliably. I suggest /thinker/today/$USER/${progname} and we are expected to clean files older than 24 hours in /thinker/today

2. how long the temp file can stay -- if use 'today' it implies it can stay for 24 hours. we can let them live longer if we are lazy but if we delete it after 24 hours nobody should complain.

BTW, /tmp is not good because the name does not look beautiful and there can potentially be collisions and security issues there.
Find all posts by this user
Quote this message in a reply
10-24-2017, 11:38 AM
Post: #7
RE: Node local temporary files/dirs
(10-24-2017 11:01 AM)lingu Wrote:  A program can never guarantee to clean its state, including temp files. But that's fine. We recommend they clean their temp files, and we implement some cleaning utility later to clean old files. The real essential agreements are:
1. where the temp files are. /thinker/local/$USER/${progname} is a bit brutal -- it means no user can store non-temp file in that dir and, then, with a mixture of temp and permanent files, no cleaning utility can work reliably. I suggest /thinker/today/$USER/${progname} and we are expected to clean files older than 24 hours in /thinker/today

2. how long the temp file can stay -- if use 'today' it implies it can stay for 24 hours. we can let them live longer if we are lazy but if we delete it after 24 hours nobody should complain.

BTW, /tmp is not good because the name does not look beautiful and there can potentially be collisions and security issues there.

Adding auto-clean or lease like mechanisms sounds good. That's how /tmp/ works for latest Linux, such as CentOS 7.

But I suggest making it a little lower level to /thinker/local/today/ as we may find 24-hour is too long or too short and create more or obsolete today. Let's call it $T for later discussion.

At the same time, The $USER under $T is not suitable for every cases, e.g. http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=5169&pid=5299 .

So for the user-specific dir, I suggest

$T/byuser/$USER/

Under $T, we may have more for across-user files like

$T/share/dssync/lock/

...

@lingu

@Xinjie pls comment too.
Visit this user's website Find all posts by this user
Quote this message in a reply
10-29-2017, 10:05 AM
Post: #8
RE: Node local temporary files/dirs
(10-24-2017 11:38 AM)zma Wrote:  
(10-24-2017 11:01 AM)lingu Wrote:  A program can never guarantee to clean its state, including temp files. But that's fine. We recommend they clean their temp files, and we implement some cleaning utility later to clean old files. The real essential agreements are:
1. where the temp files are. /thinker/local/$USER/${progname} is a bit brutal -- it means no user can store non-temp file in that dir and, then, with a mixture of temp and permanent files, no cleaning utility can work reliably. I suggest /thinker/today/$USER/${progname} and we are expected to clean files older than 24 hours in /thinker/today

2. how long the temp file can stay -- if use 'today' it implies it can stay for 24 hours. we can let them live longer if we are lazy but if we delete it after 24 hours nobody should complain.

BTW, /tmp is not good because the name does not look beautiful and there can potentially be collisions and security issues there.

Adding auto-clean or lease like mechanisms sounds good. That's how /tmp/ works for latest Linux, such as CentOS 7.

But I suggest making it a little lower level to /thinker/local/today/ as we may find 24-hour is too long or too short and create more or obsolete today. Let's call it $T for later discussion.

At the same time, The $USER under $T is not suitable for every cases, e.g. http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=5169&pid=5299 .

So for the user-specific dir, I suggest

$T/byuser/$USER/

Under $T, we may have more for across-user files like

$T/share/dssync/lock/

...

@lingu

@Xinjie pls comment too.

Any further comments?
Visit this user's website Find all posts by this user
Quote this message in a reply
10-29-2017, 04:57 PM (This post was last modified: 10-29-2017 04:58 PM by YU_Xinjie.)
Post: #9
RE: Node local temporary files/dirs
(10-24-2017 11:38 AM)zma Wrote:  Adding auto-clean or lease like mechanisms sounds good. That's how /tmp/ works for latest Linux, such as CentOS 7.

But I suggest making it a little lower level to /thinker/local/today/ as we may find 24-hour is too long or too short and create more or obsolete today. Let's call it $T for later discussion.

At the same time, The $USER under $T is not suitable for every cases, e.g. http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=5169&pid=5299 .

So for the user-specific dir, I suggest

$T/byuser/$USER/

Under $T, we may have more for across-user files like

$T/share/dssync/lock/

...

@lingu

@Xinjie pls comment too.

Overall I am fine with the idea.

I would slightly prefer "users" to "byuser".
Find all posts by this user
Quote this message in a reply
10-29-2017, 06:24 PM
Post: #10
RE: Node local temporary files/dirs
(10-24-2017 11:38 AM)zma Wrote:  But I suggest making it a little lower level to /thinker/local/today/ as we may find 24-hour is too long or too short and create more or obsolete today. Let's call it $T for later discussion.

What does 'making it a little lower level to /thinker/local/today'? You mean you want to add more subdirs under /thinker/local/today?

Quote:At the same time, The $USER under $T is not suitable for every cases, e.g. http://tab.d-thinker.org/showthread.php?...http://tab.d-thinker.org/showthread.php?tid=5169&pid=5299 .

Please specify $T or give an example. The above wording makes me feel /thinker/local/today is not welcomed, then we need to know what the new solution is. Without knowing it or an example if it, we may find later we have very different notion on the locations and that would waste time.

Quote:So for the user-specific dir, I suggest

$T/byuser/$USER/

Under $T, we may have more for across-user files like

$T/share/dssync/lock/

Note 'share' often means the data in it is accessible to other users. Sometimes it means 'the space is shared among users'. So it may lead to confusion. Depending on what the meaning is, please give it a more explicit name. Also, we need be cautious in creating hidden dirs like .* dirs. Make sure we have a good reason to hide it.
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: