[pkg] upgradepkg overwrites settings in /etc/*

No! WikiFormatting doesn't look well in ticket summary. ;)

We should probably consider using unionfs

That's probably a good idea, because we most certainly don't want to have everything in fc. Even if we did, the user still should be able to effectively use the sticky-bit trick if he wants to have his own configuration files. Having to recreate all these files everytime a package upgrade is performed is an annoying waste of time.

If this problem is dealt with in a generic manner, then moving /etc/pkg/repos.conf will become a trivial/cosmetic issue.

1. To consider: should /etc/fc be also governed by unionfs?

2. Quote from unionfs-1.3/INSTALL:

   Other known limitations:
   * Unionfs does not provide cache coherency.  What this means to you is that
     if you directly modify the lower-level branches, then Unionfs will get
     confused.  You can tell Unionfs to throw out its cache and recreate all
     of its objects (lazily), by running "uniondbg -g UNION".
   

A similar comment is contained inside unionfs-2.x as well. We should remember about that, since the our intended way to use unionfs is to let upgradepkg directly modify the underlying branch.

3. #71 should be considered before starting experimenting with unionfs since unionfs-1.x is currently supported up to 2.6.18, while unionfs-2.0 is supported starting from 2.6.20.

1. Wouldn't that be an overkill as we'll by dealing with this issue by other means?

2. ACK.

3. Hmm, maybe we should once more consider if we really want to stay with 2.6.17 in the 2.1 release?

Replying to pjf:

1. Wouldn't that be an overkill as we'll by dealing with this issue by other means?

When it comes to #134 - sure, unionfs is not going to provide this functionality and may interfere somehow with an approach we will choose for this purpose.

However, recall #159 - wouldn't the "underlying branch" be a good, natural place for making "extracted contents of *new* /etc/fc temporarily available"? But again, point 2. - if unionfs is going to be the solution - direct access to branches would have to be well-tested and rock-stable.

Any other idea? E.g. extracting /etc/* with tar -k? And maybe add an utility to revert an original config file when needed... unionfs won't be much better than that, but may cause a lot of other problems...

So, for now I'd stick to tar -k (maybe I'll do it tomorrow) and we'll test, think and talk it over.

Replying to xmxwx:

So, for now I'd stick to tar -k (maybe I'll do it tomorrow) and we'll test, think and talk it over.

No, tak -k is too simplistic. I think the right thing to do is:

• always store somewhere the original config files from the package (e.g. /etc/.original/*)
• if the contents of the current config file is the same as the previous original config file AND the sticky bit of the file is not set, then replace the config file with the new one (perserving permissions of the original file)
• if the contents of the current config file differs from the previous original config file we need to notify user about that
• if the config file was not present, we may simply add it to the "current" /etc tree
• if the config file was present and in the new package it is not, we should either delete the file (if it was not modified) or just notify the user, that it is no longer needed
• Maybe a pkg revert option (or something like that) should be provided to give the user an opportunity to revert a file to the one supplied by the package? For /etc/* it would just copy it from /etc/.original. For the other files it may e.g.:
  • check its md5sum (we can store that in /var/pkg/*/FILELIST) and if it differs, download a package and extract this one file only,
  • restore its original permissions (also from modified/upgraded /var/pkg/*/FILELIST)
  • If the file is not a part of any package we may simply unlink it (or move it to another e.g. /.trash/* tree). Launching pkg revert recursively (yes, we can add such an option) the user could clean Lintrack from all alterations
  • Maybe _this_ would an overkill and letting the user know that he can see the package-supplied config file in /etc/.original/* would be enough.
• What about upgrading non-configuration files that were changed by the user (e.g. someone's modification to pkg utility, etc.)? Probably we need not to care and replace them. However, if pkg revert and all the stuff required for that was implemented, then we could make a backup of such files in /.trash/*.

How about such approach:

1. let's don't limit our consideration only to /etc - as you said, someone may want to modify pkg, etc.
2. whenever a file has the sticky bit set, don't touch it, but always notify the user:
   • implementation notes for case when the file would be deleted:
     • use diff on FILELIST
     • output the list of files that would be deleted in form that can be easily used as arguments to `rm' (ie. no additional formatting)
   • if the file would be overwritten, save the new file under /var/pkg/<pkgname>/orig/
3. we don't have to worry about the case when a new file is added, do we? even the ticket summary says "overwrites" ;-)

Regarding implementation, I think that it may be summarized as "conditional alternative root directory option for tar". It means that whenever tar encounters situation that the file it extracts already exists and it has the sticky bit set, it should change the argument of "-C" option to "/var/pkg/<pkgname>/orig".

Why to store the latest original (ie. provided by us) package files in /var/pkg?

1. to add an option to pkg (pkg diff?) which compares the "sticky version" of a file with the original one (diff -u)
2. to add an option to pkg (pkg orig?) which overwrites the sticky version with the original one (in such case the original version in /var/pkg is deleted and chmod -t is run on the file in use)

I think that it wouldn't be feasible to store all original files anywhere in the filesystem, if we decide not to limit the "+k trick" only to /etc. What's more, chmodding -t all files in interest and doing a forced pkg upgrade should bring the user all original files he wants, anyway.

I believe that providing a simpler functionality, but more general, is better than more complex but specific.

What do you think Michał, ACK? :-)

Replying to pjf:

I think that it wouldn't be feasible to store all original files anywhere in the filesystem, if we decide not to limit the "+k trick" only to /etc. What do you think Michał, ACK? :-)

ACK. There is no point in storing all the original files in an alternative tree just like e.g. svn does. This would be needed if the user would like to know what was changed in the filesystem with respect to the original contents of the packages. Or if we wanted to notify him what changed in the config files of the original packages and what should the user probably change in his own modified version of these files to keep the consistency with the rest of the package/distribution. IMHO The benefit from storing all the original files is not worth the waste of space and the hassle of its management (i.e. all the files would have to be "installed" twice - not without a hit on the performance of pkg). If we really wanted to provide such a functionality, using unionfs would probably be the only sensible solution.

What's more, chmodding -t all files in interest and doing a forced pkg upgrade should bring the user all original files he wants, anyway.

ACK. For user's convenience we could provide e.g. pkg orig or pkg revert or pkg upgrade --original. But this would be only "cosmetic", since if the user deliberately uses chmod +t then he most probably is also aware of the need of "unsticking" it in case he would like to have his changes reverted.

1. whenever a file has the sticky bit set, don't touch it, but always notify the user: if the file would be overwritten, save the new file under /var/pkg/<pkgname>/orig/

It may be a nice feature, but it is not necessary. After the user is warned that his +t file would be overwritten, he can mv this file to sth else, repeat the upgrade, compare the files and reintroduce the modifiactions apropriately.

I think that a more important feature would be to notify the user not that +t file would be overwritten (because now overwriting happens on _every_ upgrade), but that the original file changed across the packages. One of the reasonable ways to achieve it would be to store sth like .MD5SUMS in the packages (and later move it to /var/pkg/*/MD5SUMS) or maybe just attach such information to .FILELIST, since MD5SUMS would contain (almost) the same but with md5 attached to each regular file entry. (I'm afraid that I have just reinvented a wheel: isn't this one of your old ideas, Paweł?) This way the user would be disturbed only if there is something really worth the attention. Additionally, we would have a way to obtain an information whether the current filesystem contents are modified with respect to the original package. Or... we may have just a simple hardware consistency check.

IMHO this is a simple and general functionality.

What do you think Paweł, ACK? :-)

Replying to xmxwx:

ACK. For user's convenience we could provide e.g. pkg orig or pkg revert or pkg upgrade --original. But this would be only "cosmetic", since if the user deliberately uses chmod +t then he most probably is also aware of the need of "unsticking" it in case he would like to have his changes reverted.

NACK - if he forgets about some +t file (quite probable in case of tens of routers with few custom files each), he most probably wouldn't like to run pkg upgrade again, IMHO.

1. whenever a file has the sticky bit set, don't touch it, but always notify the user: if the file would be overwritten, save the new file under /var/pkg/<pkgname>/orig/

It may be a nice feature, but it is not necessary. After the user is warned that his +t file would be overwritten, he can mv this file to sth else, repeat the upgrade, compare the files and reintroduce the modifiactions apropriately.

Notice that we have to provide a batch mode (non-interactive) for pkg, plus my comment above.

I think that a more important feature would be to notify the user not that +t file would be overwritten (because now overwriting happens on _every_ upgrade), but that the original file changed across the packages.

Good point, ACK.

One of the reasonable ways to achieve it would be to store sth like .MD5SUMS in the packages (and later move it to /var/pkg/*/MD5SUMS) or maybe just attach such information to .FILELIST

Definitely the .FILELIST option is better.

Or... we may have just a simple hardware consistency check.

;-D

What do you think Paweł, ACK? :-)

Despite my first comment, ALL-ACK, as usual ;-)

BTW:

• add support for locking package versions and repositories the package is obtained from
• improve exit codes

Milestone 2.1rc1 deleted

Started work on it