Policy 3.7.0 - /usr/lib/cgi-{bin|lib}

[Alexis Sukrieh]

[ Please follow-up to debian-webapps ]

Hi,

I was working on packaging a new version of bugzilla and when I checked my
package with lintian I discovered that /usr/lib/cgi-bin is now
deprecated by the policy standards:

W: bugzilla: file-in-usr-lib-cgi-bin usr/lib/cgi-bin/bugzilla/
N:
N: Packages shipping web server CGI files should install them in
N: /usr/lib/cgi-lib, not in /usr/lib/cgi-bin. This is done to avoid
N: conflicts with the cgi-bin script alias, which is reserved for the
N: local use of webmasters. Web servers should include /cgi-lib/ as a
N: standard ScriptAlias pointing to that directory.

I understand why this change is welcome but I'm a bit surprised because
of the following points:

I'm subscribed to the debian-webapps mailing list and have never
seen any topic about this, I'm a bit surprised such a change has not
been mentioned there.

Why have we created this list for? :-/

I think that we should document somewhere how to handle this
migration. Just changing the path /usr/lib/cgi-bin to /usr/lib/cgi-lib
in our debian/rules isn't enough, we have at least to warn the user
that he has to make sure that his webserver provides a Script Aliasing
feature from cgi-lib/ to cgi-bin/.

If this is already documented somewhere, feel free to tell me where.

I plan to do the following for the bugzilla package:

1/ Add a debconf note for notyfing the user about the location change.
2/ Provide an example configuration file that enables script-aliasing
for an apache virtual host.

That won't prevent breakages on upgrades, but at least, the user will now
what happens.

Best regards,

[Wouter Verhelst]

Post by Alexis Sukrieh
1/ Add a debconf note for notyfing the user about the location change.

Eh, no. Please don't.

Debconf notes about things that were done to follow policy are the worst
cases of debconf abuse, ever.

[Sam Morris]

Post by Alexis Sukrieh
1/ Add a debconf note for notyfing the user about the location change.

As Wouter Verhelst said, don't do this! Put an entry in the NEWS.Debian
file instead.
--
Sam Morris
http://robots.org.uk/

PGP key id 5EA01078
3412 EA18 1277 354B 991B C869 B219 7FDB 5EA0 1078

[sean finney]

hi,

Post by Alexis Sukrieh
W: bugzilla: file-in-usr-lib-cgi-bin usr/lib/cgi-bin/bugzilla/
N: Packages shipping web server CGI files should install them in
N: /usr/lib/cgi-lib, not in /usr/lib/cgi-bin. This is done to avoid
N: conflicts with the cgi-bin script alias, which is reserved for the
N: local use of webmasters. Web servers should include /cgi-lib/ as a
N: standard ScriptAlias pointing to that directory.

this is a surprising change. guess that's what i get for not being
subscribed to -policy :)

first, i don't really see what the merit is of moving files from
/usr/lib/cgi-bin to /usr/lib/cgi-lib.

second, AFAIK, /usr/lib is still the domain of dpkg/debian and not the
"local use of webmasters", so why should we have to move the files?
it seems to me that the policy *at least* should instead say:

- if it is really for the local admin's use, web servers should have
/cgi-bin point at /usr/local/lib/cgi-bin or somesuch
- /cgi-lib points at /usr/lib/cgi-bin

but i'm still grappling to understand the rationale behind why this
is a desirable thing. if the desire is to provide a way for the
local admin and packages to be able to share a script aliased directory,
i would argue this is entirely the wrong way about it. i'd be happy to
elaborate further.

Post by Alexis Sukrieh
I think that we should document somewhere how to handle this
migration. Just changing the path /usr/lib/cgi-bin to /usr/lib/cgi-lib
in our debian/rules isn't enough, we have at least to warn the user
that he has to make sure that his webserver provides a Script Aliasing
feature from cgi-lib/ to cgi-bin/.

i would argue that /usr/lib/cgi-foo is an outdated approach anyway, and
that most applications ought to be scriptaliasing /package/cgi-bin
or /cgi-bin/package to somewhere under /usr/lib/package (this would in
fact be another use for the non-existant libexec dir...). but that's
just my $0.02.

in any case, this will also require updating of configuration files and
for some systems a lot of grep+sed will need to be done if the internal
systems reference something with cgi-bin in the url. it'll also break
a lot of bookmarks, though that should really be the least of our
concerns.

a note to the debian-policy folks: you may or may not be aware that
we've done a significant amount of work regarding drafting a comprehensive
and sensible policy for web servers and web applications. i don' think
it's quite ready for submission (i think the unspoken assumption is that
we'd submit it for etch+1, and after webapps-common was introduced into
the archive), but your feedback on what we've come up with so far would
be very much appreciated:

http://webapps-common.alioth.debian.org/draft/html/

for any of you who are going to be at debconf, we're hosting
a BoF session on all this too.

sean

--

[Joey Hess]

Post by sean finney
this is a surprising change. guess that's what i get for not being
subscribed to -policy :)

Not really, it was last discussed on -policy in 2003, so being
subscribed wouldn't have helped, I'm as suprised as you are.

Post by sean finney
but i'm still grappling to understand the rationale behind why this
is a desirable thing. if the desire is to provide a way for the
local admin and packages to be able to share a script aliased directory,
i would argue this is entirely the wrong way about it. i'd be happy to
elaborate further.

The rationalle is explained in #32263, though the logs are

Post by sean finney

-----
Most people setting up a web site expect /cgi-bin/ to be available for
scripts on their site. Unfortunately, Debian uses this for those scripts
packages that get installed. These two need to be independant.
As such, Debian's system needs to be altered a bit. I recommend using
instead the name "/cgi-lib/" for scripts under /usr/lib/cgi-bin/. This
will keep both features independant and not affect the general use of
the system.

Why this change was accepted into policy in 2006 when the last message
about it was back in 2003, I have no clue. All that seems to have
changed in between is the slight support for it that existed in 2003
bitrotting away.

Despite the "policy documents existing practice" mantra, and despite
indications in the bug log that bugs were being filed and web servers
being updated in 2003 to support the cgi-lib scriptalias, as of now I
can't find any web servers that actually support it or packages that use
it:

- apache2, the most used web server in Debian, doesn't support cgi-lib.
Of course, apache2 was not in as wide use in 2003.
- apache contains no references to cgi-lib and seems to not support it
either.
- boa, despite having a changelog entry about supporting cgi-lib in the
default config, and despite including the empty directory in the deb,
doesn't actually support it by default.
- I can't find a single file shipped in /usr/lib/cgi-lib in all of
Debian.
- Some bugs that mention the directory, possibly not complete, but
probably most of them:
#167509 (cern-httpd; bug was closed when it was removed from debian)
#167510 (aolserver; bug was fixed, but package is no longer in etch)
#167511 (boa; badly fixed as mentioned above)
#167512 (roxen; bug was fixed, although possibly not in the way
policy intended, based on changelog, but package no longer
in debian)
#167513 (apache; tagged wontfix since 2003)

To all indications, shipping any cgis in cgi-lib is premature and so was
the acceptance of this policy amendment.

Post by sean finney
i would argue that /usr/lib/cgi-foo is an outdated approach anyway, and
that most applications ought to be scriptaliasing /package/cgi-bin
or /cgi-bin/package to somewhere under /usr/lib/package (this would in
fact be another use for the non-existant libexec dir...). but that's
just my $0.02.

Debian still has web servers other than apache2 in it, so that seems
difficult to do.

[sean finney]

hey joey (et al),

Post by Joey Hess

Post by sean finney
this is a surprising change. guess that's what i get for not being
subscribed to -policy :)

Not really, it was last discussed on -policy in 2003, so being
subscribed wouldn't have helped, I'm as suprised as you are.

okay, so i'm not totally out of the loop then.

Post by Joey Hess
The rationalle is explained in #32263, though the logs are

Post by sean finney

-----
Most people setting up a web site expect /cgi-bin/ to be available for
scripts on their site. Unfortunately, Debian uses this for those scripts
packages that get installed. These two need to be independant.
As such, Debian's system needs to be altered a bit. I recommend using
instead the name "/cgi-lib/" for scripts under /usr/lib/cgi-bin/. This
will keep both features independant and not affect the general use of
the system.

wow. at first i thought you dropped a digit off the bug number :)

Post by Joey Hess
Why this change was accepted into policy in 2006 when the last message
about it was back in 2003, I have no clue. All that seems to have
changed in between is the slight support for it that existed in 2003
bitrotting away.

truly confounding. istr there being an equally crufty bug about wanting
to split up /usr/lib/cgi-bin to include a /usr/share too, but can't seem
to find it now.

Post by Joey Hess

Post by sean finney
i would argue that /usr/lib/cgi-foo is an outdated approach anyway, and
that most applications ought to be scriptaliasing /package/cgi-bin
or /cgi-bin/package to somewhere under /usr/lib/package (this would in
fact be another use for the non-existant libexec dir...). but that's
just my $0.02.

Debian still has web servers other than apache2 in it, so that seems
difficult to do.

i'd disagree. this is something i'd been mulling over for a while, and
i have something somewhat concrete that i think is fairly reasonable.
if policy wrt to cgi-bin is going to be changed, i would suggest:

- that cgi-bin is defined to be a location outside of debian packages'
reach entirely (/srv/www/cgi-bin or /var/www/cgi-bin, or whatever).
- httpds which support scriptaliasing ship this as the default location
- httpds which can not scriptalias it somewhere else (those that hard-code
it at compile time, i'm guessing > 0 may do this) use that as the location.
- applications which wish to provide cgi-bin based scripts are allowed
to use the scriptalias function of applicable httpds to claim
subdirectories of cgi-bin.
- under no circumstances are packages to place files in the default
cgi-bin location.
- it is the admin's privilege/responsibility/authority to modify the
contents of the default cgi-bin location.

with this approach, the admin is free to do whatever he/she wishes with
the cgi-bin directory (place files, symlink to directories provided
by debian packages, etc), without interference from debian packages.
there is also a clear distinction of domain between the local admin
and the debian package management system, which is generally a good
thing and something we seem to like doing in debian.

for the vast majority of users, packages are still able to install and
configure themselves with the local webserver, so the overall effect of
such a change would be minimal. in fact, even finding the list of
affected packages would be a fairly trivial thing to do. using
apt-file or similar, you could find all packages providing files
under /usr/lib/cgi-bin (packages providing their cgi-files outside
of this directory are already scriptaliasing). of those packages,
you could check for which ones do not explictly provide scriptalias
functionality in their configuration, and open corresponding bugreports.



sean


--

[Joey Hess]

Post by sean finney
- that cgi-bin is defined to be a location outside of debian packages'
reach entirely (/srv/www/cgi-bin or /var/www/cgi-bin, or whatever).
- httpds which support scriptaliasing ship this as the default location
- httpds which can not scriptalias it somewhere else (those that hard-code
it at compile time, i'm guessing > 0 may do this) use that as the location.
- applications which wish to provide cgi-bin based scripts are allowed
to use the scriptalias function of applicable httpds to claim
subdirectories of cgi-bin.
- under no circumstances are packages to place files in the default
cgi-bin location.
- it is the admin's privilege/responsibility/authority to modify the
contents of the default cgi-bin location.

AFAIK apache2 is the only web server package that allows scriptaliases
to be added to it in a policy conformant way (by dropping config file
snippets into /etc/apache2/conf.d/. Other web servers that support
scriptalias, like boa, centralise it all in a single conffile, which
other packages are not allowed to edit. That's why I said that there
being more web servers than apache2 in Debian is a problem.

Post by sean finney
with this approach, the admin is free to do whatever he/she wishes with
the cgi-bin directory (place files, symlink to directories provided
by debian packages, etc), without interference from debian packages.
there is also a clear distinction of domain between the local admin
and the debian package management system, which is generally a good
thing and something we seem to like doing in debian.

Of course using /cgi-lib/ for debian's cgis and /cgi-bin/ for the admin
also draws a similarly clear distinction, although the naming of
/cgi-lib/ could be clearer (as was mentioned in the policy proposal).

[sean finney]

hey joey,

Post by Joey Hess
AFAIK apache2 is the only web server package that allows scriptaliases
to be added to it in a policy conformant way (by dropping config file
snippets into /etc/apache2/conf.d/. Other web servers that support
scriptalias, like boa, centralise it all in a single conffile, which
other packages are not allowed to edit. That's why I said that there
being more web servers than apache2 in Debian is a problem.

aha. i wasn't worrying about it too much from the "configuring the
web server" aspect of things. but imho such an ability should
be provided by the httpds if the feature is desired.

furthermore, i'm wary of binaries belonging to potentially
unconfigured webapps being generally accessible for execution (which is
a problem with the current setup as well).

Post by Joey Hess

Post by sean finney
with this approach, the admin is free to do whatever he/she wishes with
the cgi-bin directory (place files, symlink to directories provided
by debian packages, etc), without interference from debian packages.
there is also a clear distinction of domain between the local admin
and the debian package management system, which is generally a good
thing and something we seem to like doing in debian.

Of course using /cgi-lib/ for debian's cgis and /cgi-bin/ for the admin
also draws a similarly clear distinction, although the naming of
/cgi-lib/ could be clearer (as was mentioned in the policy proposal).

but this doesn't actually require that files be moved around to
accomplish this, and further i'd still say that it's conceptually buggy
to have the admin messing around with files in /usr/lib. you could
equivalently accomplish the desire of the current amendment by
scriptaliasing /cgi-lib to /usr/lib/cgi-bin and /cgi-bin to
somewhere more sensible. but i'd still disagree with this being
the right way to do.



sean


--

[Stephen Gran]

Post by sean finney
hi,

Post by Alexis Sukrieh
W: bugzilla: file-in-usr-lib-cgi-bin usr/lib/cgi-bin/bugzilla/
N: Packages shipping web server CGI files should install them in
N: /usr/lib/cgi-lib, not in /usr/lib/cgi-bin. This is done to avoid
N: conflicts with the cgi-bin script alias, which is reserved for the
N: local use of webmasters. Web servers should include /cgi-lib/ as a
N: standard ScriptAlias pointing to that directory.

this is a surprising change. guess that's what i get for not being
subscribed to -policy :)
first, i don't really see what the merit is of moving files from
/usr/lib/cgi-bin to /usr/lib/cgi-lib.

This is, IMHO, a very awkward, to say the least, change. There are
currently at a rough guess:
***@gashuffer:~$ apt-file search cgi-bin | awk -F: '{print $1}' | sort -u | wc -l
135

more than a few packages using cgi-bin. Most of the httpds Debian ships
are not trivially modifiable (no run parts directories like the
apaches). And the benefit is, what? Web developers can write
unhindered to /usr/lib? Sorry?

It seems that more and more 'cgi' programs are moving away from using
cgi-bin anyway, and that as time goes on, this will be a non-issue. I
know that certainly as a policy decision at most sites I administer, I
disable direct access to /usr/lib/cgi-bin, precisely because I don't
like newly installed but unconfigured packages being web accessable.

So, we now have 135 RC bugs, plus one more for each noncompliant httpd.
Oh, well.