dir-or-file-in-var-www on single-HTML file web "apps" or the like

[Stefano Zacchiroli]

Have a read of
http://webapps-common.alioth.debian.org/draft/html/ch-httpd.html

Thanks for the pointer, I gave it a go.

I'm looking at some of the dir-or-file-in-var-www bugs reported by
Manoj. A recurrent pattern is that of packages that ship, as their
static content, a single .html file, usually a form for triggering some
CGI app. All of them reported to be buggy store that file as
/var/www/file.html.

Reading §5.3 of the above link, I wonder whether the following solution
would be appropriate:

- ship under /etc/apache2/conf.d/ a snippet with an Alias dir mapping
the package name to a dir containing the static content (a single html
file, usually)
- add an index.html -> file.html symlink in that dir

For the end user, the change would be from accessing the app as
http://localhost/file.html to accessing it as http://localhost/package/.

Would that be appropriate?

Also, can we do anything better---still remaining FHS-compatible---than
documenting that in a NEWS.Debian entry to ease transition to the new
URL?

TIA,
Cheers.

[John H. Robinson, IV]

Reading ?5.3 of the above link, I wonder whether the following solution
- ship under /etc/apache2/conf.d/ a snippet with an Alias dir mapping
the package name to a dir containing the static content (a single html
file, usually)
- add an index.html -> file.html symlink in that dir

We have webservers other than Apache.

% aptitude search -F %p '~Phttpd'|wc -l
22

Only 4 of those are Apache. apache2-mpm-(event|itk|prefork|worker)

[sean finney]

Post by John H. Robinson, IV

Reading ?5.3 of the above link, I wonder whether the following solution
- ship under /etc/apache2/conf.d/ a snippet with an Alias dir mapping
the package name to a dir containing the static content (a single html
file, usually)
- add an index.html -> file.html symlink in that dir

We have webservers other than Apache.

and many of these other web servers provide a feature similar to Alias.
with respect to those that don't, it's been previously discussed and
deemed okay for them not to work out of the box, which is a relatively
small cost for the benefit of FHS compliance and general safety/sanity.


sean

[sean finney]

hi zack,

Post by Stefano Zacchiroli
Reading §5.3 of the above link, I wonder whether the following solution
- ship under /etc/apache2/conf.d/ a snippet with an Alias dir mapping
the package name to a dir containing the static content (a single html
file, usually)
- add an index.html -> file.html symlink in that dir
For the end user, the change would be from accessing the app as
http://localhost/file.html to accessing it as http://localhost/package/.
Would that be appropriate?
Also, can we do anything better---still remaining FHS-compatible---than
documenting that in a NEWS.Debian entry to ease transition to the new
URL?

You can also Alias a path to a file (i.e. keeping things fairly transparent to
the user), though i'd argue that moving it to the latter would be better in the
long run.


sean

[Stefano Zacchiroli]

[ adding -policy to Cc: ]

Uhm, why postpone this so long? I'd hope we could find a consensus quite soon.
Then, we might not be able to fix _all_ web apps until squeeze, but at least
tthose few with dir-or-file-in-var-www :-)

I see it a tad more complicate than you, let's hope its me
overestimating the task :-)

- the agreement actually should not come among web app maintainers, but
rather among web *server* maintainers: they should agree over a
specific dir and change the default configuration of the web server so
that that dir is the document root (for the default vhost, for web
servers supporting vhosts)

* possibly, migrating to that would require offering migration paths
to package users

- then you might start migrating web apps packages so that they install
(static) stuff under that dir, preserving the per-package path as
detailed in the webapps-common policy

- then, the rule should go into policy (possibly under §9.1.1, has an
exception to FHS, not sure about the section though) and that can't
happen before due to the usual practice-should-predate-policy

If it were me to try to achieve this, I would go for a DEP to keep track
of consensus, ... but no, I'm not willing to drive this, at least not
now :-)

Cheers.

[Gunnar Wolf]

Post by Stefano Zacchiroli

Uhm, why postpone this so long? I'd hope we could find a consensus quite soon.
Then, we might not be able to fix _all_ web apps until squeeze, but at least
tthose few with dir-or-file-in-var-www :-)

I see it a tad more complicate than you, let's hope its me
overestimating the task :-)
- the agreement actually should not come among web app maintainers, but
rather among web *server* maintainers: they should agree over a
specific dir and change the default configuration of the web server so
that that dir is the document root (for the default vhost, for web
servers supporting vhosts)
* possibly, migrating to that would require offering migration paths
to package users

That's easy, as there is fewer of us than web app maintainers. And it
is a first step. We might even have a transitional symlink making
/var/www point to /var/lib/www or whatnot?

Post by Stefano Zacchiroli
- then you might start migrating web apps packages so that they install
(static) stuff under that dir, preserving the per-package path as
detailed in the webapps-common policy
- then, the rule should go into policy (possibly under §9.1.1, has an
exception to FHS, not sure about the section though) and that can't
happen before due to the usual practice-should-predate-policy
If it were me to try to achieve this, I would go for a DEP to keep track
of consensus, ... but no, I'm not willing to drive this, at least not
now :-)

It is a bit ambitious to have it _completely_ done by Squeeze. But we
can start pushing the right way. And anyway, I do not feel it is
asking too much. Yes, we cannot just go from using /var/www to
having its existence violate policy and making insta-RC, but as soon
as the (major at least) webserves change their defaults, we can start
filing wishlist bugs, pointing maintainers to this being a work in
progress and expecting it to be strictly enforced by policy for
squeeze+1.

[Stefano Zacchiroli]

I don't get it. This would of course solve the problem of FHS compliance
but apart from that it doesn't gain anything, does it?
Now, do I totally misunderstand the issue here, or are we just moving
the /var/www problem to /var/lib/www?

No, actually we gain, but it was poorly explained (I've surely
contributed to that).

What I was aiming to is a kind of document root which is under full
control of the package manager; hence where the sysadm cannot touch
anything by hand. That's actually the only way to have a meaningful
package-based namespace.

The property I'd like is that if a package drops static files there
under a dir like package/, then those files automagically appear (in the
default vhost) as http://localhost/package/.

Now, written this way, it looks harder to implement, because it is a
kind of overlay over a default docroot (unless we assume the default
docroot is read-only for the sysadm, which doesn't look reasonable).

I see two solution to this:

- either the final URL is something like
http://localhost/debian/package/ (replace "debian" with whatever
prefix you like) and have some other per-web-server mechanism take
care of adding an Alias/link/whatever from "debian" to that new static
doc root

- or we add some kind of postinst-time registration mechanism (with all
the usual drawbacks) that check that new static doc root and register
every (new) dir there as Alias for the installed web servers. Assuming
that web servers can register themselves to the mechanism, that would
also solve the problem that webapp maintainers not necessarily have
the knowledge of the syntax of all available web servers

Any other idea?

Many thanks for sharpening the analysis, Jan.

Cheers.

PS please keep the -webapps Cc:, I believe it is truly relevant to this
thread

[Jan Hauke Rahm]

I'm commenting a bit between the paragraphs to sharpen my mind :)

Post by Stefano Zacchiroli
What I was aiming to is a kind of document root which is under full
control of the package manager; hence where the sysadm cannot touch
anything by hand. That's actually the only way to have a meaningful
package-based namespace.

I agree. The files installed by the package manager must be seperated
from user/admin files.

Post by Stefano Zacchiroli
The property I'd like is that if a package drops static files there
under a dir like package/, then those files automagically appear (in the
default vhost) as http://localhost/package/.

That would mean dropping all files in whatever dir is configured (in all
available web servers) as DocRoot.

Post by Stefano Zacchiroli
Now, written this way, it looks harder to implement, because it is a
kind of overlay over a default docroot (unless we assume the default
docroot is read-only for the sysadm, which doesn't look reasonable).

Right, either the admin can (and will) write to the DocRoot (which was
meant to be reserved for the package manager), or we need two DocRoots
configured which I don't think is possible with common web servers.

Post by Stefano Zacchiroli
- either the final URL is something like
http://localhost/debian/package/ (replace "debian" with whatever
prefix you like) and have some other per-web-server mechanism take
care of adding an Alias/link/whatever from "debian" to that new static
doc root

Like all packages drop their files in /var/lib/www/package and some
default package (or the web server itself) drops a symlink
/var/www/debian -> /var/lib/www ?
This would again lead to at least one file name in the sysadmin's space
that he/she can't use which is what we try to avoid in the first place.

Post by Stefano Zacchiroli
- or we add some kind of postinst-time registration mechanism (with all
the usual drawbacks) that check that new static doc root and register
every (new) dir there as Alias for the installed web servers.

That is basically what happens today in many cases, except that the
files aren't dropped in one location but rather in /var/cache/$package
or similar. The package's postinst registers the new files/dirs with the
web servers (usually just apache2 and/or lighttpd).

Post by Stefano Zacchiroli
Assuming that web servers can register themselves to the mechanism,
that would also solve the problem that webapp maintainers not
necessarily have the knowledge of the syntax of all available web
servers

You mean like a trigger that is dealt with by any web server installed?

Post by Stefano Zacchiroli
Any other idea?

Actually, I don't see a way to implement that (but then I'm probably not
the best code writer). It would be really nice if the web servers could
somehow register themselves for this but I think in the end this is
really what webapps-common was aimed at, isn't it? Except the
self-registering of web servers webapps-common could solve this all. A
package build-depends on it and gets provided with necessary
post{inst,rm} lines to register with known web servers. Considering that
there are that many web servers in the archive, it should be managable
for webapps-common maintainers to keep up2date with those.

So, after all I think we need some manpower in webapps-common and we
have a solution. Or not?

Post by Stefano Zacchiroli
Many thanks for sharpening the analysis, Jan.

You're welcome, although I prefer to be called Hauke. I'm a
middle-name-user ;)

Post by Stefano Zacchiroli
PS please keep the -webapps Cc:, I believe it is truly relevant to
this thread

Ack, sorry for dropping it before.

Hauke

[Stefano Zacchiroli]

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli
Now, written this way, it looks harder to implement, because it is a
kind of overlay over a default docroot (unless we assume the default
docroot is read-only for the sysadm, which doesn't look reasonable).

Right, either the admin can (and will) write to the DocRoot (which was
meant to be reserved for the package manager), or we need two DocRoots
configured which I don't think is possible with common web servers.

... or we find a technical way to enforce a kind of overlay, which is
what the two alternative solutions I advanced try to do.

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli
- either the final URL is something like
http://localhost/debian/package/ (replace "debian" with whatever
prefix you like) and have some other per-web-server mechanism take
care of adding an Alias/link/whatever from "debian" to that new static
doc root

Like all packages drop their files in /var/lib/www/package and some
default package (or the web server itself) drops a symlink
/var/www/debian -> /var/lib/www ?
This would again lead to at least one file name in the sysadmin's space
that he/she can't use which is what we try to avoid in the first place.

Well, everything is not black or white, we are not really forcing
anything on the sysadm. First of all it is just one file and only for a
single vhost in the default configuration (not in all possible
configuration the sysadm might prepare later on), and finally for all
web servers that support Alias-like directives it is not even a file: it
is just a config line that can be easily commented out.

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli
Assuming that web servers can register themselves to the mechanism,
that would also solve the problem that webapp maintainers not
necessarily have the knowledge of the syntax of all available web
servers

You mean like a trigger that is dealt with by any web server installed?

Yes, exactly.

Post by Jan Hauke Rahm
Actually, I don't see a way to implement that (but then I'm probably not
the best code writer). It would be really nice if the web servers could
somehow register themselves for this but I think in the end this is
really what webapps-common was aimed at, isn't it? Except the

I believe so. If we go forward with that, that would be a first step
toward uniforming the packaging of webapps, whether it is called
webapps-common or not I personally couldn't care less.

Post by Jan Hauke Rahm
self-registering of web servers webapps-common could solve this all. A
package build-depends on it and gets provided with necessary
post{inst,rm} lines to register with known web servers. Considering that
there are that many web servers in the archive, it should be managable
for webapps-common maintainers to keep up2date with those.

No, not really. With webservers registering themselves I was thinking at
something like each maintainer of a webserver package providing a script
(which a specified API which is the same for all webservers). That
script will be in charge of mapping one generic Alias directive (coming
from a single webapp package) to the way of implementing that directive
in the specific webserver. This way the webapps-common infrastructure is
no longer in charge of maintaining neither the knowledge nor the
implementation of every single webserver, it is only in charge of
maintaining the generic infrastructure.

Post by Jan Hauke Rahm
So, after all I think we need some manpower in webapps-common and we
have a solution. Or not?

That's out of doubt :-)

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli
Many thanks for sharpening the analysis, Jan.

You're welcome, although I prefer to be called Hauke. I'm a
middle-name-user ;)

Ooops, sorry :-/

Cheers.

[Jan Hauke Rahm]

Post by Stefano Zacchiroli

Post by Jan Hauke Rahm
self-registering of web servers webapps-common could solve this all. A
package build-depends on it and gets provided with necessary
post{inst,rm} lines to register with known web servers. Considering that
there are that many web servers in the archive, it should be managable
for webapps-common maintainers to keep up2date with those.

No, not really. With webservers registering themselves I was thinking at
something like each maintainer of a webserver package providing a script
(which a specified API which is the same for all webservers). That
script will be in charge of mapping one generic Alias directive (coming
from a single webapp package) to the way of implementing that directive
in the specific webserver. This way the webapps-common infrastructure is
no longer in charge of maintaining neither the knowledge nor the
implementation of every single webserver, it is only in charge of
maintaining the generic infrastructure.

Okay, I understand. Now, I see two ways actually to solve this.

1. If we have a generic location for packages to drop their
html/php/whatever files, like /var/lib/www, all web servers can keep
their DocRoot as /var/www and provide an alias for /var/lib/www, for
instance /debian. That way webapp packages don't even have to care
about the web server in charge, we don't need a webapps-common, we
just need all web servers to provide that alias (or if they can't,
they have to symlink it). Every webapp would be available at
localhost/debian/webapp. Since it's either a symlink or a conffile
that makes this /debian alias, it can be changed by the local
sysadmin without any risks and without touching our packages data.
2. If however all packages put their files in different locations, we
need your suggested solution with scripts for each webserver. A
package like webapps-common could run
foreach server in /usr/share/webapps-common/webservers/*; do
./$server webappPath webappName
done
and the web servers provide aliases/symlinks accordingly.

So, what's wrong with (1) that we don't go this simple path?

Hauke

[Stefano Zacchiroli]

Post by Jan Hauke Rahm
Okay, I understand. Now, I see two ways actually to solve this.
1. If we have a generic location for packages to drop their
html/php/whatever files, like /var/lib/www, all web servers can keep
their DocRoot as /var/www and provide an alias for /var/lib/www, for
instance /debian. That way webapp packages don't even have to care
about the web server in charge, we don't need a webapps-common, we
just need all web servers to provide that alias (or if they can't,
they have to symlink it). Every webapp would be available at
localhost/debian/webapp. Since it's either a symlink or a conffile
that makes this /debian alias, it can be changed by the local
sysadmin without any risks and without touching our packages data.
2. If however all packages put their files in different locations, we
need your suggested solution with scripts for each webserver. A
package like webapps-common could run
foreach server in /usr/share/webapps-common/webservers/*; do
./$server webappPath webappName
done
and the web servers provide aliases/symlinks accordingly.
So, what's wrong with (1) that we don't go this simple path?

Fair question. As I can't came up with an answer, I guess that (1) is
indeed a better solution, due Occam's razor.

I only have a couple of remarks on some details:

- According to FHS, /var/lib/ is for "variable state information". As we
are talking about static HTML content, which only change upon package
upgrade, I believe it would not be appropriate. A better place would
probably be /usr/share/www/PACKAGE/ (maybe some FHS guru can give us
some insights here ...)

- Regarding the URL that would be mapped to that dir, I don't
particularly like /debian/ (even though I've advanced it). However the
alternative solutions I can come up with (e.g. /packages/) are
actually uglier. So I guess /debian/ can stay. Some of the -webapps
people can probably come up with wiser suggestions ...

Cheers.

[Harald Braumann]

On Thu, 5 Nov 2009 16:39:06 +0100

Post by Stefano Zacchiroli
- Regarding the URL that would be mapped to that dir, I don't
particularly like /debian/ (even though I've advanced it). However
the alternative solutions I can come up with (e.g. /packages/) are
actually uglier. So I guess /debian/ can stay. Some of the -webapps
people can probably come up with wiser suggestions ...

/debian/ seems to be the de facto standard for Debian archives. So I
guess it wouldn't be such a good idea to use it for something else
(sorry, can't really come up with a better name myself).

Cheers,
harry

[The Fungi]

Post by Harald Braumann
/debian/ seems to be the de facto standard for Debian archives. So I
guess it wouldn't be such a good idea to use it for something else
(sorry, can't really come up with a better name myself).

Something short, generic and distro-neutral like /app/ would be my
personal preference if I were developing a standard for my servers.
Unfortunately, going that direction as also increases the chances of
remapping a path the admin was already using...

[Jan Hauke Rahm]

Post by Stefano Zacchiroli

Post by Jan Hauke Rahm
Okay, I understand. Now, I see two ways actually to solve this.
1. If we have a generic location for packages to drop their
html/php/whatever files, like /var/lib/www, all web servers can keep
their DocRoot as /var/www and provide an alias for /var/lib/www, for
instance /debian. That way webapp packages don't even have to care
about the web server in charge, we don't need a webapps-common, we
just need all web servers to provide that alias (or if they can't,
they have to symlink it). Every webapp would be available at
localhost/debian/webapp. Since it's either a symlink or a conffile
that makes this /debian alias, it can be changed by the local
sysadmin without any risks and without touching our packages data.
2. If however all packages put their files in different locations, we
need your suggested solution with scripts for each webserver. A
package like webapps-common could run
foreach server in /usr/share/webapps-common/webservers/*; do
./$server webappPath webappName
done
and the web servers provide aliases/symlinks accordingly.
So, what's wrong with (1) that we don't go this simple path?

Fair question. As I can't came up with an answer, I guess that (1) is
indeed a better solution, due Occam's razor.
- According to FHS, /var/lib/ is for "variable state information". As we
are talking about static HTML content, which only change upon package
upgrade, I believe it would not be appropriate. A better place would
probably be /usr/share/www/PACKAGE/ (maybe some FHS guru can give us
some insights here ...)

Full ack, and I even like /usr/share/www. It's easy to understand and
pretty unprobable that we'd have a package called www in the archive
some day needing this location.

Post by Stefano Zacchiroli
- Regarding the URL that would be mapped to that dir, I don't
particularly like /debian/ (even though I've advanced it). However the
alternative solutions I can come up with (e.g. /packages/) are
actually uglier. So I guess /debian/ can stay. Some of the -webapps
people can probably come up with wiser suggestions ...

Manoj suggested '/vendor-apps' and I like that. It says what it should
say and doesn't steal any probable path.

I still see a problem with the upgrade path for existing installations.
I might be wrong but I think the most difficult cases are very custom
setups with lots of changes by the local admin. I'm thinking of e.g.
webmail.domain.tld being a virtual host with DocRoot
/usr/share/squirrelmail. If the files there move to
/usr/share/www/squirrelmail we break a lot of setups. So, what about
shipping a symlink from the old location to the new one as a migration
path? This doesn't solve the very default (e.g. users accessing
squirrelmail via localhost/squirrelmail) but that is so easily solved
via alias directive or symlink that I suppose a NEWS.Debian entry would
fit best here.
What do you say?

Now, I'm willing to run this, i.e. file bugs against web servers, wait
for them to be fixed, then file bugs against web applications (if
needed, I'm right now looking into a way to make a lintian check for it,
e.g. package-with-section-web-but-no-files-in-canonical-docroot). But I
don't feel like we're having a clear consensus here, do we?

Attached is a suggestion for MBF as well as a dd-list of all relevant
web servers (if I did my job right).

Now critizise!
Hauke

[Charles Plessy]

Post by Jan Hauke Rahm
I still see a problem with the upgrade path for existing installations.
I might be wrong but I think the most difficult cases are very custom
setups with lots of changes by the local admin. I'm thinking of e.g.
webmail.domain.tld being a virtual host with DocRoot
/usr/share/squirrelmail. If the files there move to
/usr/share/www/squirrelmail we break a lot of setups. So, what about
shipping a symlink from the old location to the new one as a migration
path? This doesn't solve the very default (e.g. users accessing
squirrelmail via localhost/squirrelmail) but that is so easily solved
via alias directive or symlink that I suppose a NEWS.Debian entry would
fit best here.
What do you say?

Dear Hauke,

As a maintainer of a web application, I share your worries. I never had any
user request to make it work out of the box with alternative web servers, so I
guess that my users have nothing to gain and everything to lose in a change. I
strongly suggest that the transition is experimented on a few volunteer
packages before increasing the workload of many persons – users and developers.

For new packages, grouping everything in /usr/share/www sounds like a good
idea. The alias name, « vendor », I find a bit disturbing because we do not
sell anything. But picking the name will be the priviledge of the Do-o-crat who
will lead the transition, I presume.

Still, having /usr/share/www as a document root does not prevent complex
packages to be fragmented between /usr/share, /usr/lib/cgi-bin/, /var/lib/,
/var/tmp, /var/run and /etc. Maybe you can double-check how many web servers
are able to cope with that before starting to invest a lot of time. Otherwise,
since shipping configuration files in /etc/webserver/conf.d will still be
necessary for these packages to work, there will little benefit in moving files
to /usr/share/www.

Anyway, thank you very much for your initiative. Exploring new directions is good !

Have a nice day,

[Jan Hauke Rahm]

Thanks for your response, Charles!

Post by Charles Plessy
As a maintainer of a web application, I share your worries. I never had any
user request to make it work out of the box with alternative web servers, so I
guess that my users have nothing to gain and everything to lose in a change. I
strongly suggest that the transition is experimented on a few volunteer
packages before increasing the workload of many persons – users and developers.

I think this is a bit black or white. Users (ar rather admins) gain the
possibility to easily guess where a package is to be found. No looking
for /usr/share/package/html or something, just assume it's
/usr/share/www/package. And they don't lose everything, they just need
to read the chapter (yet to be written, of course) in the release notes
about web applications being moved; additionally NEWS.Debian should be
provided by any relevant package and so on.

Also, I said, a proper migration path has still to be figured out. The
question for now is: do we want this change at all and -- if so -- shall
I file bugs against the web servers. I'd suggest severity wishlist for
the start.

Post by Charles Plessy
For new packages, grouping everything in /usr/share/www sounds like a good
idea. The alias name, « vendor », I find a bit disturbing because we do not
sell anything. But picking the name will be the priviledge of the Do-o-crat who
will lead the transition, I presume.

Well, I find 'vendor' a bit confusing as well but I'm not a native
english speaker and I don't have better ideas. Shoot if you have any.
Something like 'debian', 'webapps', 'applications' seem way to generic
for this.

Post by Charles Plessy
Still, having /usr/share/www as a document root does not prevent complex
packages to be fragmented between /usr/share, /usr/lib/cgi-bin/, /var/lib/,
/var/tmp, /var/run and /etc. Maybe you can double-check how many web servers
are able to cope with that before starting to invest a lot of time. Otherwise,
since shipping configuration files in /etc/webserver/conf.d will still be
necessary for these packages to work, there will little benefit in moving files
to /usr/share/www.

And there isn't even a way we could (or want to) solve this.
Applications of many sorts have to deal with FHS, webapps are nothing
special in that matter. We can't allow an admin to fiddle aroung with
files in /usr/share, nor can we just pump all webapps into /var as this
will break (or at least bloat) many backup strategies and cause other
problems.

Until now I thought simple symlinks work with every webserver and thus
didn't see a problem for that. It's the application that sometimes needs
patching to deal with its being splattered around. Am I wrong here?

Hauke

[Henrik Andreasson]

On Sat, 7 Nov 2009, Jan Hauke Rahm wrote:

Caudium can and will adjust to any standard that the community agrees
upon and it can handle different directories without problem.

I really dont have that much input for how this should be done but leaving
it as it is now is worse.

Post by Jan Hauke Rahm
Thanks for your response, Charles!

Post by Charles Plessy
As a maintainer of a web application, I share your worries. I never had any
user request to make it work out of the box with alternative web servers, so I
guess that my users have nothing to gain and everything to lose in a change. I
strongly suggest that the transition is experimented on a few volunteer
packages before increasing the workload of many persons ? users and developers.

I think this is a bit black or white. Users (ar rather admins) gain the
possibility to easily guess where a package is to be found. No looking
for /usr/share/package/html or something, just assume it's
/usr/share/www/package. And they don't lose everything, they just need
to read the chapter (yet to be written, of course) in the release notes
about web applications being moved; additionally NEWS.Debian should be
provided by any relevant package and so on.
Also, I said, a proper migration path has still to be figured out. The
question for now is: do we want this change at all and -- if so -- shall
I file bugs against the web servers. I'd suggest severity wishlist for
the start.

Post by Charles Plessy
For new packages, grouping everything in /usr/share/www sounds like a good
idea. The alias name, « vendor », I find a bit disturbing because we do not
sell anything. But picking the name will be the priviledge of the Do-o-crat who
will lead the transition, I presume.

Well, I find 'vendor' a bit confusing as well but I'm not a native
english speaker and I don't have better ideas. Shoot if you have any.
Something like 'debian', 'webapps', 'applications' seem way to generic
for this.

Post by Charles Plessy
Still, having /usr/share/www as a document root does not prevent complex
packages to be fragmented between /usr/share, /usr/lib/cgi-bin/, /var/lib/,
/var/tmp, /var/run and /etc. Maybe you can double-check how many web servers
are able to cope with that before starting to invest a lot of time. Otherwise,
since shipping configuration files in /etc/webserver/conf.d will still be
necessary for these packages to work, there will little benefit in moving files
to /usr/share/www.

And there isn't even a way we could (or want to) solve this.
Applications of many sorts have to deal with FHS, webapps are nothing
special in that matter. We can't allow an admin to fiddle aroung with
files in /usr/share, nor can we just pump all webapps into /var as this
will break (or at least bloat) many backup strategies and cause other
problems.
Until now I thought simple symlinks work with every webserver and thus
didn't see a problem for that. It's the application that sometimes needs
patching to deal with its being splattered around. Am I wrong here?
Hauke

[Stefano Zacchiroli]

Post by Charles Plessy
For new packages, grouping everything in /usr/share/www sounds like a good
idea. The alias name, « vendor », I find a bit disturbing because we do not
sell anything. But picking the name will be the priviledge of the Do-o-crat who
will lead the transition, I presume.

Well, it is actually pretty common in cross-distro lingo, Debian is a
vendor as well as pretty much every other distro is. The advantage of
settling on such a name IMO would be a higher chance in making it
popular in other distros. Also, it is a name that I _think_ is pretty
unlike to be used by local admins.

Post by Charles Plessy
Still, having /usr/share/www as a document root does not prevent complex
packages to be fragmented between /usr/share, /usr/lib/cgi-bin/, /var/lib/,
/var/tmp, /var/run and /etc. Maybe you can double-check how many web servers
are able to cope with that before starting to invest a lot of time. Otherwise,
since shipping configuration files in /etc/webserver/conf.d will still be
necessary for these packages to work, there will little benefit in moving files
to /usr/share/www.

I don't understand this argument. Sure, complex packages will be split
in several dirs, our policy states the rule for that to happen. The
whole point of this standardization is to have a single URL prefix under
which _entry_points_ for shipped web applications can be found, no
matter how the applications are deployed on the filesystem. I found such
a goal worthwhile by itself and orthogonal to the other concern you
raise.

Cheers.

[Charles Plessy]

Post by Stefano Zacchiroli

Post by Charles Plessy
Still, having /usr/share/www as a document root does not prevent complex
packages to be fragmented between /usr/share, /usr/lib/cgi-bin/, /var/lib/,
/var/tmp, /var/run and /etc. Maybe you can double-check how many web servers
are able to cope with that before starting to invest a lot of time. Otherwise,
since shipping configuration files in /etc/webserver/conf.d will still be
necessary for these packages to work, there will little benefit in moving files
to /usr/share/www.

I don't understand this argument. Sure, complex packages will be split
in several dirs, our policy states the rule for that to happen. The
whole point of this standardization is to have a single URL prefix under
which _entry_points_ for shipped web applications can be found, no
matter how the applications are deployed on the filesystem. I found such
a goal worthwhile by itself and orthogonal to the other concern you
raise.

Hi Stefano,

the lintian error dir-or-file-in-var-www exists for a long time, and I believe
that most packages with active maintainers have already been split according to the
FHS. What I question is whether it is worth the effort to move the content of
/usr/share/<package> to /usr/share/www/<package>:

- How many purely static websites do we distribute as Debian packages?
(Note that /usr/share/doc/<package> is already served as http://localhost/doc/package/)

- How many dynamic websites will start to work out of the box without
the need for a specific configuration for each webserver?

I checked at the web application I maintain (emboss-explorer), and in its
particular case, it would still need an apache.conf file. That is not enough to
make statistics, so I am just asking if there will be many packages that can
take advantage of the proposed reorganisation. [And unfortunately source.debian.net
looks borken again…].

Of course, if the use of /usr/share/www/<package> is optional, everybody wins.

Cheers,
--
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

[Stefano Zacchiroli]

Post by Charles Plessy
the lintian error dir-or-file-in-var-www exists for a long time, and I
believe that most packages with active maintainers have already been
split according to the FHS. What I question is whether it is worth the
effort to move the content of /usr/share/<package> to
- How many purely static websites do we distribute as Debian packages?
(Note that /usr/share/doc/<package> is already served as http://localhost/doc/package/)
- How many dynamic websites will start to work out of the box without
the need for a specific configuration for each webserver?

I now understand better your argument, thanks for rephrasing. I don't
have an answer, because I haven't done the test, but I do agree that it
would be interesting to know in advance. Still, it is not exactly clear
to me how to test this, how would you automatically discover whether a
package has a static splash screen (note indeed that it is not only
about "purely static" web applications, but also about "regular"
webapps, with a static splash screen).

Post by Charles Plessy
I checked at the web application I maintain (emboss-explorer), and in its
particular case, it would still need an apache.conf file. That is not enough to
make statistics, so I am just asking if there will be many packages that can
take advantage of the proposed reorganisation. [And unfortunately source.debian.net
looks borken again
].

Can you perhaps explain why so?

I frankly hope that with /vendor/ + /usr/lib/cgi-bin/ (which we already
have), and maybe with some symlinks under /vendor/ we will be able to
address quite a lot of issues. It would be interesting to known which
one we can't.

Now that I think of it, probably a per-package data dir would help, but
that can be a tad more tricky due to single-instance of
multiple-instance nature of the webapp in question ...

Post by Charles Plessy
Of course, if the use of /usr/share/www/<package> is optional,
everybody wins.

It is forcibly optional: if you have some static content you should use,
otherwise not.

Cheers.

[sean finney]

Post by Stefano Zacchiroli
I frankly hope that with /vendor/ + /usr/lib/cgi-bin/ (which we already
have), and maybe with some symlinks under /vendor/ we will be able to
address quite a lot of issues. It would be interesting to known which
one we can't.

objection, you honor! :)

something that hasn't really been brought up (i mentioned it on the
non-webapps thread in -devel already) is that this makes packages
potentially opened in an unconfigured state. unless you can ensure that
the system is only running on localhost, it has some significant security
implications. personally i'd rather that /usr/lib/cgi-bin goes the way
of the dodo, and that packages are required to ship/generate webserver
config files if they want to function out of the box.


sean

[Russ Allbery]

Post by sean finney
something that hasn't really been brought up (i mentioned it on the
non-webapps thread in -devel already) is that this makes packages
potentially opened in an unconfigured state. unless you can ensure that
the system is only running on localhost, it has some significant
security implications. personally i'd rather that /usr/lib/cgi-bin goes
the way of the dodo, and that packages are required to ship/generate
webserver config files if they want to function out of the box.

Wholeheartedly agreed, particularly if we can put a management system in
place similar to the (really nice) Apache module management system that
lets admins selectively enable specific applications, which installing
everything into a default CGI-active directory doesn't permit as easily.

[Jan Hauke Rahm]

Post by Russ Allbery

Post by sean finney
something that hasn't really been brought up (i mentioned it on the
non-webapps thread in -devel already) is that this makes packages
potentially opened in an unconfigured state. unless you can ensure that
the system is only running on localhost, it has some significant
security implications. personally i'd rather that /usr/lib/cgi-bin goes
the way of the dodo, and that packages are required to ship/generate
webserver config files if they want to function out of the box.

Wholeheartedly agreed, particularly if we can put a management system in
place similar to the (really nice) Apache module management system that
lets admins selectively enable specific applications, which installing
everything into a default CGI-active directory doesn't permit as easily.

Not that I'm opposing to what you're saying but... every application in
the archive is configured during the installation process, possibly
asking debconf questions, providing defaults etc. After the installation
it should run in a mode that suites most use cases and is secure. We (or
at least I) always expected that.

Now with web applications, if I read you suggestions correctly, you want
to just throw the files in the system, leave it unconfigured without
meaningfull defaults, even leading to an unsecure state, and then blame
the web server for not securing the application?

Or am I misunderstanding you?
Hauke

[sean finney]

hi jan,

Post by Jan Hauke Rahm
Not that I'm opposing to what you're saying but... every application in
the archive is configured during the installation process, possibly
asking debconf questions, providing defaults etc. After the installation
it should run in a mode that suites most use cases and is secure. We (or
at least I) always expected that.

i think there's some ambiguity due to different uses of the term
configuration... i meant largely wrt the application configuration,
not the package's status wrt dpkg (though the latter probably also
implies the former).

Post by Jan Hauke Rahm
Now with web applications, if I read you suggestions correctly, you want
to just throw the files in the system, leave it unconfigured without
meaningfull defaults, even leading to an unsecure state, and then blame
the web server for not securing the application?

the issue that i'm raising is that in this proposal the default "vendor"
docroot is on by default, and the existing cgi-bin directory has
files that are executed by the webserver without regard to whether the
application is (or is properly) configured. therefore it will be activated
as soon as the files are unpacked and the onus is then placed on the
packager and (more likely) the local admin to make sure that this window
of oppurtunity is closed with appropriate webserver configuration after
the package has been installed.

given that there is likely a requirement for real world applications
to register themselves with a webserver anyway (for things which couldn't
"just work", extra Alias's for FHS splitting, etc), i don't see the
argument for making some subset of the package active when the larger
problem still exists and it potentially exposes the system security wise
in the meantime.

i'm not arguing against having standards and a system for getting apps to
work out of the box, in fact that's why this list was created in teh first
place. but much of this discussion is treading over already trodden paths :)


sean
--

[Stefano Zacchiroli]

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli

Post by Jan Hauke Rahm
1. If we have a generic location for packages to drop their
html/php/whatever files, like /var/lib/www, all web servers can keep
their DocRoot as /var/www and provide an alias for /var/lib/www, for
instance /debian. That way webapp packages don't even have to care
about the web server in charge, we don't need a webapps-common, we
just need all web servers to provide that alias (or if they can't,
they have to symlink it). Every webapp would be available at
localhost/debian/webapp. Since it's either a symlink or a conffile
that makes this /debian alias, it can be changed by the local
sysadmin without any risks and without touching our packages data.

- According to FHS, /var/lib/ is for "variable state information". As we
are talking about static HTML content, which only change upon package
upgrade, I believe it would not be appropriate. A better place would
probably be /usr/share/www/PACKAGE/ (maybe some FHS guru can give us
some insights here ...)

Full ack, and I even like /usr/share/www. It's easy to understand and
pretty unprobable that we'd have a package called www in the archive
some day needing this location.

Fine, it seems we're done with this aspect then.

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli
- Regarding the URL that would be mapped to that dir, I don't
particularly like /debian/ (even though I've advanced it). However the
alternative solutions I can come up with (e.g. /packages/) are
actually uglier. So I guess /debian/ can stay. Some of the -webapps
people can probably come up with wiser suggestions ...

Manoj suggested '/vendor-apps' and I like that. It says what it should
say and doesn't steal any probable path.

Right, but it is possibly hard to type and a bit ugly, I've
counter-proposed "/vendor/" (see my separate mail on that).

Post by Jan Hauke Rahm
I still see a problem with the upgrade path for existing installations.
I might be wrong but I think the most difficult cases are very custom
setups with lots of changes by the local admin. I'm thinking of e.g.
webmail.domain.tld being a virtual host with DocRoot
/usr/share/squirrelmail. If the files there move to
/usr/share/www/squirrelmail we break a lot of setups. So, what about
shipping a symlink from the old location to the new one as a migration
path? This doesn't solve the very default (e.g. users accessing
squirrelmail via localhost/squirrelmail) but that is so easily solved
via alias directive or symlink that I suppose a NEWS.Debian entry would
fit best here.
What do you say?

I think that migrations from complex setup to this new setup will stay
complex no matter what we do. Also, it is not really something we can
"standardize" upon as migrations are very specific to each involved
packages and will ultimately be dealed with by single maintainers. So
I'd refrain to propose a generic upgrade path and just describe the new
situation we want to obtain.

Post by Jan Hauke Rahm
Now, I'm willing to run this, i.e. file bugs against web servers, wait
for them to be fixed, then file bugs against web applications (if
needed, I'm right now looking into a way to make a lintian check for it,
e.g. package-with-section-web-but-no-files-in-canonical-docroot). But I
don't feel like we're having a clear consensus here, do we?

Well, defining consensus is always a tricky business :), but I haven't
heard significant voices against, am I wrong? I'd personally proceed as
follow: write a draft document (even a very brief one!) which summarizes
the proposal so that people do not need to dig into the thread to follow
the evolution. Once we have it, re-post it to the relevant lists (I'd
say -devel, -policy, -webapps) and ask for comments.

Actually, your suggested MBF text can pretty much be that document. If
it were me, I'd open a DEP on that just because I fear losing track of
it, but YMMV.

Post by Jan Hauke Rahm
Attached is a suggestion for MBF as well as a dd-list of all relevant
web servers (if I did my job right).

Wow, thanks!

Cheers.

[Jan Hauke Rahm]

Post by Stefano Zacchiroli

Post by Jan Hauke Rahm
I still see a problem with the upgrade path for existing installations.
I might be wrong but I think the most difficult cases are very custom
setups with lots of changes by the local admin. I'm thinking of e.g.
webmail.domain.tld being a virtual host with DocRoot
/usr/share/squirrelmail. If the files there move to
/usr/share/www/squirrelmail we break a lot of setups. So, what about
shipping a symlink from the old location to the new one as a migration
path? This doesn't solve the very default (e.g. users accessing
squirrelmail via localhost/squirrelmail) but that is so easily solved
via alias directive or symlink that I suppose a NEWS.Debian entry would
fit best here.
What do you say?

I think that migrations from complex setup to this new setup will stay
complex no matter what we do. Also, it is not really something we can
"standardize" upon as migrations are very specific to each involved
packages and will ultimately be dealed with by single maintainers. So
I'd refrain to propose a generic upgrade path and just describe the new
situation we want to obtain.

I agree, I was just pointing out that common setups can have a proper
migration path. We could give a hint when we're at it but the maintainer
needs to think of something him/herself after all.

Post by Stefano Zacchiroli

Post by Jan Hauke Rahm
Now, I'm willing to run this, i.e. file bugs against web servers, wait
for them to be fixed, then file bugs against web applications (if
needed, I'm right now looking into a way to make a lintian check for it,
e.g. package-with-section-web-but-no-files-in-canonical-docroot). But I
don't feel like we're having a clear consensus here, do we?

Well, defining consensus is always a tricky business :), but I haven't
heard significant voices against, am I wrong? I'd personally proceed as
follow: write a draft document (even a very brief one!) which summarizes
the proposal so that people do not need to dig into the thread to follow
the evolution. Once we have it, re-post it to the relevant lists (I'd
say -devel, -policy, -webapps) and ask for comments.

I'll try to come up with something within the next 24 hours. Don't know
yet if it's gonna be a DEP or just another mail but summarizing what we
got so far sounds like a plan.

Hauke

[Stefan Fritsch]

Post by Jan Hauke Rahm

Post by Stefano Zacchiroli

Post by Jan Hauke Rahm
Now, I'm willing to run this, i.e. file bugs against web
servers, wait for them to be fixed, then file bugs against web
applications (if needed, I'm right now looking into a way to
make a lintian check for it, e.g.
package-with-section-web-but-no-files-in-canonical-docroot).
But I don't feel like we're having a clear consensus here, do
we?

Well, defining consensus is always a tricky business :), but I
haven't heard significant voices against, am I wrong? I'd
personally proceed as follow: write a draft document (even a very
brief one!) which summarizes the proposal so that people do not
need to dig into the thread to follow the evolution. Once we have
it, re-post it to the relevant lists (I'd say -devel, -policy,
-webapps) and ask for comments.

I'll try to come up with something within the next 24 hours. Don't
know yet if it's gonna be a DEP or just another mail but
summarizing what we got so far sounds like a plan.

I think it would be best to have a wiki page that lists the problems
you are trying to solve and also possible pitfalls that have to be
taken into account. Then it is easier to check if a proposed solution
would solve the problems while not breaking anything else.

One of the pitfalls is apache2-suexec. It has the document root
compiled in and doesn't allow more than one document root.

[Neil McGovern]

Post by Stefano Zacchiroli
- the agreement actually should not come among web app maintainers, but
rather among web *server* maintainers: they should agree over a
specific dir and change the default configuration of the web server so
that that dir is the document root (for the default vhost, for web
servers supporting vhosts)

For reference, when the initial RFH on a webapps policy was sent out,
all httpd maintainers were included.

Post by Stefano Zacchiroli
If it were me to try to achieve this, I would go for a DEP to keep track
of consensus, ... but no, I'm not willing to drive this, at least not
now :-)

I personally don't like the DEPs, but would be happy to accept reasoned
patches into the webapps policy submission.

Neil

[Giacomo A. Catenazzi]

Post by Stefano Zacchiroli
[ adding -policy to Cc: ]

Uhm, why postpone this so long? I'd hope we could find a consensus quite soon.
Then, we might not be able to fix _all_ web apps until squeeze, but at least
tthose few with dir-or-file-in-var-www :-)

I see it a tad more complicate than you, let's hope its me
overestimating the task :-)
- the agreement actually should not come among web app maintainers, but
rather among web *server* maintainers: they should agree over a
specific dir and change the default configuration of the web server so
that that dir is the document root (for the default vhost, for web
servers supporting vhosts)
* possibly, migrating to that would require offering migration paths
to package users
- then you might start migrating web apps packages so that they install
(static) stuff under that dir, preserving the per-package path as
detailed in the webapps-common policy
- then, the rule should go into policy (possibly under §9.1.1, has an
exception to FHS, not sure about the section though) and that can't
happen before due to the usual practice-should-predate-policy

Personally I would like to have a competently different approach:

- web server ask where to put the root (probably proposing default
a /srv/www location). But not further assumption about the location.
Admins, per FHS, could choose other paths.

This could be done by a new update-http-root application.
(and ev. could handle multiple vhost).
And possibly allowing no public location (thus forcing local only
connections): we tend to forget about this, but IMHO more and more
desktop computers are installed with webserver because of local
convenience. Thus we really need to securely support this common
cases.


- No webapp are installed "live" by default:

We have too much crap web application, and some/most of our users
don't realize that they are installing a public accessible crap.
[the desktop users]

Thus IMHO we need a "update-webappl" utility, which would
list, ask and ev. install the just installed webapplication.

This is not so far as the installation of apache modules, which
ask for which apache (apache/apache2/apache2-ssl/...) to enable
modules. We just list the possible web root.

Naturally admins can skip this point (e.g. not allowing debian
to handle webappl, but doing manually).

Probably a webserver-specific support script will handle the
generation of symlink (default) or via configuration (webserver
specific) of the /usr/lib/cgi or /usr/share/* dir.


In short:

- no hardcoded default root location (only a default value for a
real user question)
- not installing by default (without asking) web apps.


ciao
cate

PS: first mail in debian-policy, so maybe I missed the point of
the discussion (which take place in the other mailing lists)

[Jan Hauke Rahm]

Post by Giacomo A. Catenazzi
- web server ask where to put the root (probably proposing default
a /srv/www location). But not further assumption about the
location.
Admins, per FHS, could choose other paths.

This is done in a few webapps right now. But we (as in maintainers)
still need to provide a sane (FHS and policy comliant) default for
non-interactive installations and even for users who have no clue about
what they're doing and simply press enter.

Post by Giacomo A. Catenazzi
This could be done by a new update-http-root application.
(and ev. could handle multiple vhost).
And possibly allowing no public location (thus forcing local only
connections): we tend to forget about this, but IMHO more and more
desktop computers are installed with webserver because of local
convenience. Thus we really need to securely support this common
cases.

Well, if we go for a solution that implements vhosts (in case of apache
and other web servers that even know about vhosts) we could include some
command to bind the vhost to localhost. I'd be okay with it. If,
however, we decide to have one debian specific DocRoot which all webapps
whould use, then we don't have vhosts for all packages but one generic
one. We can still have this per default bound to localhost but if one
webapp is public, all are.

Post by Giacomo A. Catenazzi
We have too much crap web application, and some/most of our users
don't realize that they are installing a public accessible crap.
[the desktop users]

Isn't that what the topic was about? *confused*

Post by Giacomo A. Catenazzi
Thus IMHO we need a "update-webappl" utility, which would
list, ask and ev. install the just installed webapplication.
This is not so far as the installation of apache modules, which
ask for which apache (apache/apache2/apache2-ssl/...) to enable
modules. We just list the possible web root.
Naturally admins can skip this point (e.g. not allowing debian
to handle webappl, but doing manually).
Probably a webserver-specific support script will handle the
generation of symlink (default) or via configuration (webserver
specific) of the /usr/lib/cgi or /usr/share/* dir.

What exactly should these scripts do then?

Post by Giacomo A. Catenazzi
- no hardcoded default root location (only a default value for a
real user question)

As said above, we need a DocRoot that is FHS and policy compliant. If I
understand you correctly you want a DocRoot for each and every webapp
instead of one global DocRoot?

Post by Giacomo A. Catenazzi
- not installing by default (without asking) web apps.

You mean a debconf template that says: "I know you typed 'aptitude
install webapp' but we're smarter than you are and thus not doing it
unless you really want to and type 'yes' now." ? Remember Debian is not
only about a secure system but also about a working system. Don't make
life harder than necessary. I'd agree to binding to localhost, though.

Hauke