Piwigo, Owncloud, ...: doing it not right?

[Stefan Monnier]

[ I originally started this discussion in the freedombox context. ]

I've been maintaining "home servers" for many years now, first using
OpenWRT and then Debian.

Maintaining those boxes with Debian is a lot easier than with OpenWRT,
thanks to the great job done by the Debian project: the
unattended-upgrades package takes care of the main work, and an
occasional "apt-get upgrade" takes care of the rest.

But there's always been a sore point: it seems that packages that I could
describe as "cloud service", like owncloud, piwigo, gitlab and friends
are rarely available in Debian, and when they are it tends to be
short lived (after taking a long time to appear).

IIUC this is due to the enormous amount of work needed to massage the
upstream package so that it fits well within the Debian system, abiding
by the various conventions, etc...

So at this point I'm wondering is it would make sense to have another
repository (a bit like `experimental`) to hold quick&dirty packages
which are "done wrong", i.e. where the packaging effort is minimal (not
good enough to make it into sid).

This comes from my experience with the piwigo and owncloud packages,
which were nicely packaged in Debian, but that took a fair bit of effort
and in the end I'm not sure if that effort was worthwhile. The benefits
to me when I installed the package didn't pay off in the long term when
I had to convert to the non-Debian-packaged (let's call it "upstream")
version:
- overall, I think I (as a user) would have been better served by
a "poorly integrated" package which stayed closer to the upstream
(the disadvantage is bit more effort at the beginning, but
compensated by availability of more docs since I can rely more on the
upstream docs, and if the Debian package disappears, conversion back
to upstream would be much easier).
- by reducing the effort on the Debian packager side, I increase the
likelihood that a package will be available and will be kept available
(and uptodate) for a long time.

So I'm wondering what people think here: would there be room for another
repository (call it `messy`) which would hold extra packages that are
not well integrated with the rest of Debian. I'm not sure if those
packages should be installable on top of stable or testing or both
(probably stable is the most important).

Of course, I wouldn't want such a repository to encourage maintainers to
give up on nicely integrated packages. The intention would be for this
repository to hold applications which otherwise wouldn't be in Debian at
all (of course, all those packages would have to be compatible the DFSG).


Stefan

[Tomasz Muras]

Post by Stefan Monnier
[ I originally started this discussion in the freedombox context. ]
I've been maintaining "home servers" for many years now, first using
OpenWRT and then Debian.
Maintaining those boxes with Debian is a lot easier than with OpenWRT,
thanks to the great job done by the Debian project: the
unattended-upgrades package takes care of the main work, and an
occasional "apt-get upgrade" takes care of the rest.
But there's always been a sore point: it seems that packages that I could
describe as "cloud service", like owncloud, piwigo, gitlab and friends
are rarely available in Debian, and when they are it tends to be
short lived (after taking a long time to appear).
IIUC this is due to the enormous amount of work needed to massage the
upstream package so that it fits well within the Debian system, abiding
by the various conventions, etc...
So at this point I'm wondering is it would make sense to have another
repository (a bit like `experimental`) to hold quick&dirty packages
which are "done wrong", i.e. where the packaging effort is minimal (not
good enough to make it into sid).
This comes from my experience with the piwigo and owncloud packages,
which were nicely packaged in Debian, but that took a fair bit of effort
and in the end I'm not sure if that effort was worthwhile. The benefits
to me when I installed the package didn't pay off in the long term when
I had to convert to the non-Debian-packaged (let's call it "upstream")
- overall, I think I (as a user) would have been better served by
a "poorly integrated" package which stayed closer to the upstream
(the disadvantage is bit more effort at the beginning, but
compensated by availability of more docs since I can rely more on the
upstream docs, and if the Debian package disappears, conversion back
to upstream would be much easier).
- by reducing the effort on the Debian packager side, I increase the
likelihood that a package will be available and will be kept available
(and uptodate) for a long time.
So I'm wondering what people think here: would there be room for another
repository (call it `messy`) which would hold extra packages that are
not well integrated with the rest of Debian. I'm not sure if those
packages should be installable on top of stable or testing or both
(probably stable is the most important).
Of course, I wouldn't want such a repository to encourage maintainers to
give up on nicely integrated packages. The intention would be for this
repository to hold applications which otherwise wouldn't be in Debian at
all (of course, all those packages would have to be compatible the DFSG).

To make it short and not repeat what you've said already: based on my
experience
with packaging Moodle, I agree with everything you've said.
In that case it's either your "upstream, messy" package or nothing.

Thanks for bringing that up,
Tomek

[Paul Wise]

Post by Stefan Monnier
So I'm wondering what people think here: would there be room for another
repository (call it `messy`) which would hold extra packages that are
not well integrated with the rest of Debian. I'm not sure if those
packages should be installable on top of stable or testing or both
(probably stable is the most important).

Sounds like you are waiting for the Debian bikesheds/PPAs proposal to
be implemented. Until then, you could create a messy.debian.net
repository for this.

Personally, it seems like a weird idea to me. What would be the
advantage of messy Debian packages over whatever system upstream has
in place for installation and upgrades? I thought that usually those
are fairly well automated using web interfaces already? Some upstreams
already have repos for messy Debian packages (piwik for eg).

In any case, I think the general movement upstream is away from distro
packaging and towards more-standardised upstream-provided "apps" in
various forms (Docker/Flatpak/snappy/etc). If you want messy, I think
using the existing messes is better than spending effort on creating
messy Debian packages.

Post by Stefan Monnier
Of course, I wouldn't want such a repository to encourage maintainers to
give up on nicely integrated packages. The intention would be for this
repository to hold applications which otherwise wouldn't be in Debian at
all (of course, all those packages would have to be compatible the DFSG).

I think you should drop the DFSG requirement, because then you have to
deal with DFSG item 2, which means tracking down source code for each
item. Web stuff often uses compiled HTML/JS/CSS and soon WebAssembly
and dealing with that is a big part of packaging messy web apps.
Documenting and checking copyright and license information would also
be required, which is another big part of this. The third aspect is
embedded code copies; with automated conversion of many
language-specific packages to Debian packages, that work could be
reduced.

https://wiki.debian.org/AutomaticPackagingTools

[Nicolas]

Hi all,

I was maintainer for piwigo package for many years. It helps a lot as I was
also a core member team.
I cannot precisely why I stopped manage that package but I can remember
acceptance rules are more and more constrainant. I think about minified
stuff (javascript, stylesheets) that need to be unminified in debian.
Upstream not always understand problem. And it gives maintainers more work
to minified them afterwards to have a good visitors experience. Flash is
another big problem. I didn't find a solution even if flash's end of life
is more and more real !

I don't think using a messy server is a good idea. Working on active
project and follow upstream release is not so difficult. The first release
is a bit difficult but afterwards it can be done easily.

My two cents.
Nicolas

Post by Paul Wise

Post by Stefan Monnier
So I'm wondering what people think here: would there be room for another
repository (call it `messy`) which would hold extra packages that are
not well integrated with the rest of Debian. I'm not sure if those
packages should be installable on top of stable or testing or both
(probably stable is the most important).

Sounds like you are waiting for the Debian bikesheds/PPAs proposal to
be implemented. Until then, you could create a messy.debian.net
repository for this.
Personally, it seems like a weird idea to me. What would be the
advantage of messy Debian packages over whatever system upstream has
in place for installation and upgrades? I thought that usually those
are fairly well automated using web interfaces already? Some upstreams
already have repos for messy Debian packages (piwik for eg).
In any case, I think the general movement upstream is away from distro
packaging and towards more-standardised upstream-provided "apps" in
various forms (Docker/Flatpak/snappy/etc). If you want messy, I think
using the existing messes is better than spending effort on creating
messy Debian packages.

Post by Stefan Monnier
Of course, I wouldn't want such a repository to encourage maintainers to
give up on nicely integrated packages. The intention would be for this
repository to hold applications which otherwise wouldn't be in Debian at
all (of course, all those packages would have to be compatible the DFSG).

I think you should drop the DFSG requirement, because then you have to
deal with DFSG item 2, which means tracking down source code for each
item. Web stuff often uses compiled HTML/JS/CSS and soon WebAssembly
and dealing with that is a big part of packaging messy web apps.
Documenting and checking copyright and license information would also
be required, which is another big part of this. The third aspect is
embedded code copies; with automated conversion of many
language-specific packages to Debian packages, that work could be
reduced.
https://wiki.debian.org/AutomaticPackagingTools
--
bye,
pabs
https://wiki.debian.org/PaulWise

[Paul Wise]

Post by Nicolas
I was maintainer for piwigo package for many years.
I cannot precisely why I stopped manage that package

Looking at the removal bug, it sounds like there wasn't time to make a
non-messy package; it was removed for lack of response to RC bugs,
security issues, missing dependencies, embedded code copies, non-free
content, upgrade issues etc.

https://bugs.debian.org/694820
https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=both;dist=unstable;package=piwigo

Post by Nicolas
acceptance rules are more and more constrainant. I think about minified
stuff (javascript, stylesheets) that need to be unminified in debian.

These requirements have always been there for all of Debian, they
haven't become more restrictive over time. It was just that people
packaging web apps didn't think about the requirements and then the
rest of Debian noticed that was happening and started working on QA
efforts to automatically detect these problems and file bugs so that
maintainers would fix them.

Post by Nicolas
Upstream not always understand problem. And it gives maintainers more work
to minified them afterwards to have a good visitors experience.

It should be simple for Debian to use the same automated minification
process as upstream does right?

Post by Nicolas
Flash is another big problem. I didn't find a solution even if flash's end of life is
more and more real !

Flash is dead, it is time for everyone to stop supporting it. If your
packages build-depend on Flash related things, I strongly suggest
removing them from upstream.

[Nicolas]

Post by Paul Wise

Post by Nicolas
I was maintainer for piwigo package for many years.
I cannot precisely why I stopped manage that package

Looking at the removal bug, it sounds like there wasn't time to make a
non-messy package; it was removed for lack of response to RC bugs,
security issues, missing dependencies, embedded code copies, non-free
content, upgrade issues etc.
https://bugs.debian.org/694820
https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=
both;dist=unstable;package=piwigo

I know that and I'm not very confortable with theses bugs. Some can be
resolve upstream. Others depends on others package.

Post by Paul Wise

Post by Nicolas
acceptance rules are more and more constrainant. I think about minified
stuff (javascript, stylesheets) that need to be unminified in debian.

These requirements have always been there for all of Debian, they
haven't become more restrictive over time. It was just that people
packaging web apps didn't think about the requirements and then the
rest of Debian noticed that was happening and started working on QA
efforts to automatically detect these problems and file bugs so that
maintainers would fix them.

Yes it's the point. But as I said it can be done but it's more work for
maintainers but it gives a better debian release.

Post by Paul Wise

Post by Nicolas
Upstream not always understand problem. And it gives maintainers more

work

Post by Nicolas
to minified them afterwards to have a good visitors experience.

It should be simple for Debian to use the same automated minification
process as upstream does right?

Not so simple. For some files yes but some others are merged.

Post by Paul Wise

Post by Nicolas
Flash is another big problem. I didn't find a solution even if flash's

end of life is

Post by Nicolas
more and more real !

Flash is dead, it is time for everyone to stop supporting it. If your
packages build-depend on Flash related things, I strongly suggest
removing them from upstream.

I know but if upstream does not remove flash how as a maintainer can I do ?
It was the main issue for me that makes me stop working on that package.
Flash is used to make upload multiple. It can be done in html5 but upstream
don't to it completely. So I didn't know how to do it.

Don't misunderstand me, I don't reject fault. It's mainly my fault. I try
to do my best to maintain that package but it was more and more stuff.

Nicolas

Post by Paul Wise
https://wiki.debian.org/PaulWise

[Paul Wise]

Post by Nicolas
Not so simple. For some files yes but some others are merged.

What do you mean by "others are merged"?

Post by Nicolas
I know but if upstream does not remove flash how as a maintainer can I do ?

Same as for any migration; investigate what Flash is used for and if
HTML5 can do the same thing. If it can, file a bug report asking for
addition of HTML5 support and wait for that to be done, send a patch
to speed it up. If HTML5 can't do what Flash is used for, lobby the
browser vendors to implement a replacement and go back to your
upstream.

Post by Nicolas
It was the main issue for me that makes me stop working on that package.
Flash is used to make upload multiple. It can be done in html5 but upstream
don't to it completely. So I didn't know how to do it.

It looks very very simple to support multi-file uploads now, it seems
to be a single HTML <input> tag attribute:

https://www.w3schools.com/TAGS/att_input_multiple.asp

I would guess that there are JavaScript based fallbacks too.

Post by Nicolas
Don't misunderstand me, I don't reject fault. It's mainly my fault. I try to
do my best to maintain that package but it was more and more stuff.

I wouldn't blame you for the state of web development, for that I
blame upstreams :)

[Romain Beauxis]

Hi all,

Post by Nicolas
I was maintainer for piwigo package for many years. It helps a lot as I was
also a core member team.
I cannot precisely why I stopped manage that package but I can remember
acceptance rules are more and more constrainant. I think about minified
stuff (javascript, stylesheets) that need to be unminified in debian.
Upstream not always understand problem. And it gives maintainers more work
to minified them afterwards to have a good visitors experience. Flash is
another big problem. I didn't find a solution even if flash's end of life

is

Post by Nicolas
more and more real !
I don't think using a messy server is a good idea. Working on active

project

Post by Nicolas
and follow upstream release is not so difficult. The first release is a

bit

Post by Nicolas
difficult but afterwards it can be done easily.

I feel compelled to add my two cents here. I joined this mailing list
probably about 10 years ago. Back then, I wanted to promote a set of
packaging helpers to help simlink back and forth the different pieces of a
webapps depending on where they'd ought to be, /usr/share, /var/lib etc..
And, yeah, it was still a mess.

I've become convinced that the current constrains for splitting files is a
case of the cathedral syndrome: some skilled people build a beautiful
cathedral but then we loose the knowledge about the whys and hows and just
keep doing the same without understanding why.

Some of the constrains about the file system are outdated. Splitting files
in /usr/share because, in the old days, mainframe servers would share these
accros stations with different architectures? That sounds like a 12th
century cathedral now. Android file system has been completely overhauled
and, with docker containers, for instance, this is only going to go more
into that direction.

More generally, even though I understand the idea of wanting to maintain a
sane, clean and secured state for the global system, when the packaging
system gets too much in the way of the user, then you have a fundamental
conflict as to which is helping which. That goes for the minifed files that
have to be un-minified and etc.

Some major languages are already progressively phasing out of the main
packaging framework, ruby with the gems, Python with pip, OCaml with opam
and etc. It's unfortunate but perhaps this is the next century's way of
building cathedrals?

Romain

[Christopher Huhn]

Hi

Post by Paul Wise
Personally, it seems like a weird idea to me. What would be the
advantage of messy Debian packages over whatever system upstream has
in place for installation and upgrades?

I'm packaging a few packages for internal use at our institute, some of
them web apps, recently eg. opennetadmin
<https://github.com/chuhn/ona/tree/debian>

Post by Paul Wise
In any case, I think the general movement upstream is away from distro
packaging and towards more-standardised upstream-provided "apps" in
various forms (Docker/Flatpak/snappy/etc). If you want messy, I think
using the existing messes is better than spending effort on creating
messy Debian packages.

From my point of view there are several levels of messiness (from `curl
… | sudo bash` to a fully compliant Debian package):
* Upstream web app installers are often interactive-only.
* These installers almost never keep track of installed files and
modifications.
* There's seldom an uninstall procedure etc.

You get all of this more or less automatically by turning the stuff into
a Debian package.
Additionally you get access to decent tools like dh, dbconfig-common,
Apache-integration, dependency management, log rotation aso.

Then you can iteratively make the package more Debian compliant (esp.
FHS conforming file system layout). This will help co-workers to figure
out what is where more easily.

In terms of work vs. utility from my perspective this adolescent
packaging state is sort of a sweet spot.

Getting from here to a 'proper' (first of all lintian-clean) Debian
package is normally a lot of detective work: Understand several build
systems, investigate where code and embedded stuff comes from,
acknowledge the right copyrights, fix compatibility issues when
replacing embedded libs with their Debian-provided counterparts (where
either one may be outdated …)

This would also be the point to file an ITP bug - which I personally
hesitated to do during my 15 years with Debian (shame on me).

I think a repo for immature packages might be a good thing. It could
also attract upstreams that are interested at providing Debian packages
but don't have the proper understanding of the principles behind the
Debian distro.

But this repo should be more like an incubator:
* Starting with first-shot messy packages
* Attract experienced packagers and interested users
* Add feedback from automated packaging checkers and qa tools
* Finally: Make package fit for Debian main

Does that make sense to you?
Once upon a time, unstable was a place for such activities. But AFAICT
maturity constraints for packages to enter unstable have indeed become
more and more strict (or strictly applied).

Kind regards

Christopher

[Stefan Monnier]

Post by Paul Wise
Personally, it seems like a weird idea to me. What would be the
advantage of messy Debian packages over whatever system upstream has
in place for installation and upgrades? I thought that usually those
are fairly well automated using web interfaces already? Some upstreams
already have repos for messy Debian packages (piwik for eg).

The upstream installation process is usually different for each app, and
often sucks (web interfaces for update? really?). Sometimes I feel
like the upstream assumes this app is an important part of your life so
you can afford to take time to follow its development (i.e. it's not
meant for a personal server but for a company-wide server managed by
a professional sysadmin).

Many offer Debian packages, but having a Debian repository instead has
the following advantages:
- one additional repository in my sources.list for N apps, rather than
N additional repositories.
- at least a tiny bit of third-party oversight (I'm willing to give
some trust to upstream(s), but some third party oversight is very
welcome).
- the packager is aware of (and hopefully agrees with) Debian's
philosophy, so the package will presumably make some effort to reduce
friction with Debian. As Christopher Huhn says, I'd expect it to work
as an incubator.

Post by Paul Wise
In any case, I think the general movement upstream is away from distro
packaging and towards more-standardised upstream-provided "apps" in
various forms (Docker/Flatpak/snappy/etc).

Not familiar with Flatpak nor Snappy, but what I've seen of Docker is
not promising in terms of long-term maintenance, since the way to keep
an app up-to-date is rather painful: seems to work well for
a professional sysadmin managing a company-wide installation, but for
myself managing N apps for personal use this looks like an insane amount
of work. I haven't seen anything remotely close to "apt-get
upgrade" there.

Post by Paul Wise

Post by Stefan Monnier
all (of course, all those packages would have to be compatible the DFSG).

I think you should drop the DFSG requirement, because then you have to
deal with DFSG item 2, which means tracking down source code for each
item.

By "compatible with DFSG" I meant that it should be Free Software.

Post by Paul Wise
Web stuff often uses compiled HTML/JS/CSS and soon WebAssembly
and dealing with that is a big part of packaging messy web apps.

I wouldn't require from "messy" packages that they solve those messes.
Only that solving them be possible (i.e. that the source code be
available somewhere).

Post by Paul Wise
Documenting and checking copyright and license information would also
be required, which is another big part of this.

I think this part of the work is important, OTOH, so contrary to the
previous messes, I would think we'd want this work to be done before
a package can be accepted in the "messy" repository.

IOW, the main criteria for acceptance into "messy" would be:
- Licenses have been checked to ensure it's all Free Software.
- it doesn't mess up a normal install: while it doesn't have to follow
the FHS, it should confine its non-FHS-conforming files to
a harmless area.

Post by Paul Wise
The third aspect is embedded code copies;

Allowing embedded copies would be part of what makes packages "messy", yes.

Post by Paul Wise
with automated conversion of many language-specific packages to Debian
packages, that work could be reduced.

A "messy" package obviously wouldn't *have to* use embedded copies, so
among the "messy" packages, some would be messier than others.


Stefan

[Paul Wise]

Post by Stefan Monnier

Post by Paul Wise

Post by Stefan Monnier
all (of course, all those packages would have to be compatible the DFSG).

I think you should drop the DFSG requirement, because then you have to
deal with DFSG item 2, which means tracking down source code for each
item.

By "compatible with DFSG" I meant that it should be Free Software.

I don't think software that isn't compliant with DFSG item 2 (ie it is
missing source code for some part) but is released under a
DFSG-compatible license counts as Free Software.

Post by Stefan Monnier
I wouldn't require from "messy" packages that they solve those messes.
Only that solving them be possible (i.e. that the source code be
available somewhere).

Figuring out where the source code is and how to build it is often the
biggest part of the challenge in ensuring that things are Free
Software, mainly because automated build practices and proper
non-source file management are very much lacking in many upstreams.