On 17/05/16 09:06, Jonas Oberg wrote:
Dear Fellow,
over the next months, we'll be rolling out an idea of fellow-run services. As you know, the FSFE has been offering several services to its volunteers over the years: a blog platform, an XMPP service, email forwards, our wiki, and so on.
One of our volunteers, Florian Snow, has volunteered to coordinate a team of volunteers who will look at the possibility of taking over maintenance of our blog platform and run this as a service for other volunteers.
By forming small volunteer teams around specific services, we hope this will provide first the necessary autonomy to the individual teams to run the service as they would want it run, and second, a useful learning experience for the volunteers who take up the role of maintaining a service.
Can you comment on what this means from a technical perspective? Does autonomy mean that volunteers will be able to pick any arbitrary version of an application and start running it on FSFE hardware? Or will volunteers run it on their own hardware and if so, how will things like authentication credentials be shared?
In Debian we also have services that are maintained by individual developers, they fall into two categories:
a) *.debian.net - these services run on a Developers' own servers, debian.net DNS is the only thing managed centrally, Debian provides no guarantees about the availability of such services
b) *.debian.org - services running on Debian (DSA) managed hardware. Any applications used on such machines have to be packaged and backported to the stable release (e.g. jessie-backports). These have a higher level of availability and security updates.
People who want to ensure a service is available 24x7 can generally help by ensuring all the software and dependencies are properly packaged and backported or helping raise funds for a developer to do that work. While it seems tedious at first, this is how we achieve security and availability. For example, when the developer of a package or backport is on vacation or retires, it is relatively easy for a security team member who never saw the package before to get involved and release an official patch for it.
I often come across people who insist that they have to run the latest version of something from Git, usually with a hodge-podge of dependencies downloaded directly from different web sites. These types of arrangements are often very unique and very hard to support when the person is on holiday or decides to stop supporting it.
Regards,
Daniel
Hi Daniel,
Can you comment on what this means from a technical perspective?
I will comment with our thinking at the moment, but happy to hear thoughts and ideas, as this is something we're just starting to experiment with.
Does autonomy mean that volunteers will be able to pick any arbitrary version of an application and start running it on FSFE hardware?
I would leave the decision of which version of a particular application to run up to the volunteers who run the service, as they would be the ones who know the software best. When someone wants to start a team to run a particular service, however, our principal system administrators team will need to approve this.
Over time, I imagine us developing some more rigid guidelines for this, but so far I've only started sketching out the very basics here:
http://wiki.fsfe.org/TechDocs/FellowRunServices
Or will volunteers run it on their own hardware and if so, how will things like authentication credentials be shared?
We expect everything to run on our own hardware, which is a pre-requisite for our current authentication scheme. At the moment, services would authenticate via our LDAP, but we expect to deprecate this soon in favor of either (1) an openID connector, or (2) individual passwords for the respective services.
I often come across people who insist that they have to run the latest version of something from Git, usually with a hodge-podge of dependencies downloaded directly from different web sites. These types of arrangements are often very unique and very hard to support when the person is on holiday or decides to stop supporting it.
I agree, but I would be okay with supporting this, if the team agrees this is the way they want to run the servie. It's important to note, I believe, that all services provided in this way -- and indeed, most work of the FSFE -- is done in teams, with a one coordinator and a deputy coordinator.
We should not get into the situation where we have a single person responsible for a service, and indeed, building a single point of failure in that way doesn't contribute to the learning which we would expect either. So if we get to the point where there is no interest beyond one individual to run the service, it would likely be better to not run the service at all.
And perhaps this ought to go in our guidelines: each team should, for their service, clearly document the steps needed to be taken to shut down the service (including exporting data to the individuals, sending out shut down messages, etc).
Sincerely,
Hi Daniel,
Daniel Pocock daniel@pocock.pro writes:
Can you comment on what this means from a technical perspective? Does autonomy mean that volunteers will be able to pick any arbitrary version of an application and start running it on FSFE hardware?
Let me try to answer that for the blogs: We have not discussed the details here, so what I say might not be the final solution. The current plan is to keep the blogs on FSFE hardware and also have the operating system maintained by system hackers. That simplifies maintenance overall. If an OS update breaks the application running on top of it, that's the team's problem. That way, system hackers do not have an additional burden and the service team can take care of the application itself.
I see the appropriate solution as a mixture of the two scenarios you have at Debian: The service runs on FSFE maintained servers, but if anything, the FSFE only makes sure those servers run. If the service breaks, it's not the FSFE's problem. So no guarantee of availability for the server at all.
ensuring all the software and dependencies are properly packaged
Right now, the blogs run on Wordpress. It is fairly straightforward to install (and update within the same major version). Packaging newer versions of Wordpress would create a lot of additional work and would cause a delay in rolling out possibly important updates. Wordpress is also pretty isolated, so I think decent documentation of how it was set up should be sufficient here. But the team is still forming, so we need to still discuss it. So far, that's just my opinion.
I often come across people who insist that they have to run the latest version of something from Git, […]
I agree that that is a bad idea. I see the blogs as a service that is relatively small and we can figure out how things work best here and use that experience if and when further services run by sustaining members are set up. Ideally, I want to move away from Wordpress, but that is not a goal we will be able to achieve immediately. If we can come up with a good static solution, that will reduce depencies on external packages and simplify the setup further.
To sum this up a bit: I think you're making valid points and I will keep them in mind. For more complex setups, packaging might be useful. In this particular case, I think it's not necessary, but we will have to see what the rest of the team thinks.
Happy hacking! Florian
Hi,
Let me try to answer that for the blogs: We have not discussed the details here, so what I say might not be the final solution. The current plan is to keep the blogs on FSFE hardware and also have the operating system maintained by system hackers. That simplifies maintenance overall. If an OS update breaks the application running on top of it, that's the team's problem. That way, system hackers do not have an additional burden and the service team can take care of the application itself.
It might be a good idea to have some minimal formalized communication between server-admins and service-admins in this scenario:
1. The service documentation must include a contact so that the admin can be notified without asking around
2. The service documentation should include minimal functional tests so that server-admins can possibly detect breakage (this can be basic at first and be extended whenever something breaks in a new way)
3. The server admins should communicate (and schedule) changes in advance so that service-admins are aware
I know, that is very basic stuff and probably will be done anyway. I just wanted to point it out because in practice it *is* forgotten surprisingly often...
Cheers, Johannes
Hi Johannes,
Thank you for the input! What you write makes perfect sense and I will keep it in mind!
Happy hacking! Florian
On Wed, May 18, 2016 at 06:55:12PM +0200, Johannes Zarl-Zierl wrote: ...
- The server admins should communicate (and schedule) changes in advance so
that service-admins are aware
i think that is not realistic in all circumstances.
for any real changes, yes of course we will plan in advance (e.g. distribution release update, ...)
however, if there is a mayor security update, the lead time will most likely be quite a bit shorter, depending on the servirity (cases like hartbleed come to mind).
regards, albert
On Thursday 19 May 2016 10:18:37 Albert Dengg wrote:
On Wed, May 18, 2016 at 06:55:12PM +0200, Johannes Zarl-Zierl wrote:
- The server admins should communicate (and schedule) changes in advance
so that service-admins are aware
i think that is not realistic in all circumstances.
for any real changes, yes of course we will plan in advance (e.g. distribution release update, ...)
however, if there is a mayor security update, the lead time will most likely be quite a bit shorter, depending on the servirity (cases like hartbleed come to mind).
You're right - security updates and general emergencies usually come unannounced and can't be postponed. In my defense, I *did* think about that. I just forgot to mention it explicitly ;-)
Cheers, Johannes
On 18/05/16 18:14, Florian Snow wrote:
Hi Daniel,
Daniel Pocock daniel@pocock.pro writes:
Can you comment on what this means from a technical perspective? Does autonomy mean that volunteers will be able to pick any arbitrary version of an application and start running it on FSFE hardware?
Let me try to answer that for the blogs: We have not discussed the details here, so what I say might not be the final solution. The current plan is to keep the blogs on FSFE hardware and also have the operating system maintained by system hackers. That simplifies maintenance overall. If an OS update breaks the application running on top of it, that's the team's problem. That way, system hackers do not have an additional burden and the service team can take care of the application itself.
I see the appropriate solution as a mixture of the two scenarios you have at Debian: The service runs on FSFE maintained servers, but if anything, the FSFE only makes sure those servers run. If the service breaks, it's not the FSFE's problem. So no guarantee of availability for the server at all.
ensuring all the software and dependencies are properly packaged
Right now, the blogs run on Wordpress. It is fairly straightforward to install (and update within the same major version). Packaging newer versions of Wordpress would create a lot of additional work and would cause a delay in rolling out possibly important updates.
Wordpress is available in Debian, would the packages be suitable for you? The versions are here:
https://packages.qa.debian.org/w/wordpress.html
jessie-backports has 4.5.2
If something is in Debian then any important updates should be supported by the security team, they are usually quite fast.
Wordpress is also pretty isolated, so I think decent documentation of how it was set up should be sufficient here. But the team is still forming, so we need to still discuss it. So far, that's just my opinion.
I often come across people who insist that they have to run the latest version of something from Git, […]
I agree that that is a bad idea. I see the blogs as a service that is relatively small and we can figure out how things work best here and use that experience if and when further services run by sustaining members are set up. Ideally, I want to move away from Wordpress, but that is not a goal we will be able to achieve immediately. If we can come up with a good static solution, that will reduce depencies on external packages and simplify the setup further.
To sum this up a bit: I think you're making valid points and I will keep them in mind. For more complex setups, packaging might be useful. In this particular case, I think it's not necessary, but we will have to see what the rest of the team thinks.
I had several sites running on Drupal myself but I found that it becomes tedious dealing with PHP security bugs and such things on a regular basis. Database upgrades require some care and all of these systems have performance constraints, especially if you get slashdotted. Consequently, I moved many of the sites to a simple static hosting solution using Bootstrap and jekyll
https://github.com/jekyll/jekyll https://packages.qa.debian.org/j/jekyll.html
Jekyll transforms Markdown into HTML. The Markdown files can be stored in Git.
Regards,
Daniel
Hi Daniel,
Daniel Pocock daniel@pocock.pro writes:
Wordpress is available in Debian, would the packages be suitable for you? The versions are here:
The version does not matter so much as long as it still receives bugfixes. To be quite honest, though, I have had unpleasant experiences with Debian packages of web applications. This was several years ago and may not be accurate anymore, but back then, some of those applications had several changes made to them and that made it hard to find problems because the installation was different from most of the other installations out there.
But aside from that, I am worried about several things in this scenario: 1. An OS update always updates the Wordpress install as well. This may break necessary plugins that are not available in Debian. So that means, the system hackers would always have to check with the blog hackers before performing OS updates. I don't think this is a very good solution. 2. The Debian package does not (or at least did not) support the regular Wordpress update mechanism. That makes perfect sense from an OS package perspective, but it may cause some issues in our case here. (We might need to go through several older WP versions to get to the current one, for example, and the internal update mechanism makes that pretty easy). 3. Also, however fast the security team may be, receiving and applying the bugfix from upstream will always be faster. With publicly facing software that is known for vulnerabilities, I'd rather have updates as fast as possible. This is also pretty easy with the internal update mechanism.
Don't get me wrong, I love Debian and I am not the kind of person to use external repositories all the time or something like that, but for web applications, I tend to go with upstream. That being said, things have not been decided yet and I really appreciate your input. I will keep it in mind during my next round of tests.
I had several sites running on Drupal myself but I found that it becomes tedious dealing with PHP security bugs and such things on a regular basis.
Agreed. That is exactly my experience and the reason for looking for alternatives to Wordpress.
Consequently, I moved many of the sites to a simple static hosting solution using Bootstrap and jekyll
Thank you for mentioning this. I have set up several sites with Jekyll and Bootstrap and I am generally happy with it. There are some more modern systems that I worked with that have some advantages, for instance Pelican and Acrylamid.
However, the problem here is usability. We need to find a way to make the editing process easy for non-technical bloggers. I would imagine some of our users are more interested in the political side of Free Software and may not be hackers themselves. Finding a good solution for them as well has to be our goal. That is going to be one of the biggest issues the team will have to tackle.
Happy hacking! Florian
On 19/05/16 07:52, Florian Snow wrote:
Hi Daniel,
Daniel Pocock daniel@pocock.pro writes:
Wordpress is available in Debian, would the packages be suitable for you? The versions are here:
The version does not matter so much as long as it still receives bugfixes. To be quite honest, though, I have had unpleasant experiences
By default, every package in Debian receives security fixes. Popular web packages (like Apache, PHP and Wordpress) get these fixes within just a few hours.
This means that the system-hackers group (or any other sysadmin) can easily deploy those fixes to servers immediately using
apt-get update && apt-get upgrade
and this same command also deploys any fixes for the OS or other utilities. Having a single channel for security and bug fixes like this makes the whole process very efficient.
If a bug fix is not security critical and isn't in the stable version of a package, you can use backports.debian.org to get a newer version.
with Debian packages of web applications. This was several years ago and may not be accurate anymore, but back then, some of those applications had several changes made to them and that made it hard to find problems because the installation was different from most of the other installations out there.
These changes are usually made for some good reason and it varies a lot from one package to the next. For example, the Drupal and Redmine packages are slightly modified to make virtual hosting very easy on Debian.
But aside from that, I am worried about several things in this scenario:
- An OS update always updates the Wordpress install as well. This may break necessary plugins that are not available in Debian. So that means, the system hackers would always have to check with the blog hackers before performing OS updates. I don't think this is a very good solution.
That depends on the OS
Debian has only had a major upgrade once every two years. You don't have to upgrade immediately, the previous version is usually supported for at least one year and sometimes longer with LTS.
In any organization, it is very unusual for the sysadmin to upgrade a whole system without consent of the application manager. I would hope the system-hackers group would coordinate the date of any major OS upgrade with you.
As packaged systems are very standard, you should be able to easily replicate the server in a test environment and do a trial run of the upgrade and test any plugins before upgrading the real server.
- The Debian package does not (or at least did not) support the regular Wordpress update mechanism. That makes perfect sense from an OS package perspective, but it may cause some issues in our case here. (We might need to go through several older WP versions to get to the current one, for example, and the internal update mechanism makes that pretty easy).
You could do it this way:
a) lock the site to prevent any changes, put the database in read-only mode b) dump the database to a file c) load the database into a test server d) run through the upgrading in the test server using any non-standard procedures you require e) dump the database from the test server to a file f) load it into the real server again g) put the site back into read-write mode
You would need to test steps (b) - (d) with a trial run anyway before doing a real upgrade.
- Also, however fast the security team may be, receiving and applying the bugfix from upstream will always be faster. With publicly facing software that is known for vulnerabilities, I'd rather have updates as fast as possible. This is also pretty easy with the internal update mechanism.
Debian security updates usually come within a few hours. The Debian security team often receive private notifications of security bugs before they are advertised publicly, so they can prepare new packages in advance.
There is also the possibility of delays in the individual teams. Even if you have a great team supporting the blog platform, if FSFE has 10 different teams supporting different applications, some of the teams may not always be active all the time. Using a standard process that the system-hackers can support ensures that only the system-hackers team needs to worry about having somebody on call 24x7 for security updates.
Anybody who is concerned that the updates won't be fast enough could do a much better service to the community if they volunteered to help the Debian security team or help the system-hackers team achieve a 24x7 support roster.
Don't get me wrong, I love Debian and I am not the kind of person to use external repositories all the time or something like that, but for web applications, I tend to go with upstream. That being said, things have not been decided yet and I really appreciate your input. I will keep it in mind during my next round of tests.
I had several sites running on Drupal myself but I found that it becomes tedious dealing with PHP security bugs and such things on a regular basis.
Agreed. That is exactly my experience and the reason for looking for alternatives to Wordpress.
Consequently, I moved many of the sites to a simple static hosting solution using Bootstrap and jekyll
Thank you for mentioning this. I have set up several sites with Jekyll and Bootstrap and I am generally happy with it. There are some more modern systems that I worked with that have some advantages, for instance Pelican and Acrylamid.
However, the problem here is usability. We need to find a way to make the editing process easy for non-technical bloggers. I would imagine some of our users are more interested in the political side of Free Software and may not be hackers themselves. Finding a good solution for them as well has to be our goal. That is going to be one of the biggest issues the team will have to tackle.
Is anybody aware of a web-based tool that can be used to edit text files and commit the changes in a Git repository? That could be customized to make a web front-end to edit the blogs and some cron job could run jekyll to re-publish the site.
Maybe something like that already exists as a turn-key solution and if not, somebody could write some scripts like that and I would look at sponsoring it as a package in Debian.
Regards,
Daniel
Hi Daniel,
I really appreciate your input. However, this is getting very technical quickly, so maybe we should move these discussion to the appropriate team lists where details of implementation can be discussed. Those mailing lists could be either blog-hackers@lists.fsfe.org or system-hackers@fsfeurope.org.
Daniel Pocock daniel@pocock.pro writes:
This means that the system-hackers group (or any other sysadmin) can easily deploy those fixes to servers immediately using
apt-get update && apt-get upgrade
I understand and that is what I do on my servers. In case of the FSFE servers, this requires a bit more coordination between (now) different teams.
These changes are usually made for some good reason and it varies a lot from one package to the next.
I never doubted that. It did create additional problems for me in the past, though. Again, we will see what works best and then decide.
Debian has only had a major upgrade once every two years.
That's not what I mean; I mean regular updates. They should not break anything in Debian, I understand that. My point is just that we may have to use a Wordpress plugin (for example for LDAP) that may or may not play nice with all versions of Wordpress. In an ideal world, we would just not use such a plugin, but in reality, we may need it to provide the service we want to provide.
In the past, the system hackers installed upgrades, and, I presume, had to check if the blog was still running. We want to reduce the workload of the admins here by having a separate team take care of anything service (e.g. blog) related. The sysadmins should be able to install an OS update without having to check the blog specifically.
I would hope the system-hackers group would coordinate the date of any major OS upgrade with you.
I would hope so for a switch from Jessie to Stretch for example. For regular updates, I think no such interaction should be required. Now, the problem I was talking about is that by using a non-packaged version of Wordpress, we decouple things a bit.
The scenario with using the packaged version could look like this: A sysadmin installs updates and one of those is for Wordpress. This update breaks LDAP authentication and possibly even brings down the whole Wordpress instance. That means the blog hackers have to act immediately to fix the issue, at a time that may be inconvenient.
The scenario with upstream Wordpress could look like this: A sysadmin installs updates and Wordpress is not affected. Wordpress tries to do an automatic update and cannot do it because of the LDAP plugin. That means the blog hackers still need to act soon, but they can start when it is more convenient and when they have more time to actually track down and fix the problem.
As packaged systems are very standard, you should be able to easily replicate the server in a test environment and do a trial run of the upgrade and test any plugins before upgrading the real server.
I agree this would be ideal. I am afraid that we don't have the manpower for such a setup, though.
Is anybody aware of a web-based tool that can be used to edit text files and commit the changes in a Git repository?
That is something we need to figure out in the blog hackers team. I already looked into a lot of different solutions, but it appears there is no real backend solution out there. The problem of just editing markdown can be solved with tools that are out there, but images are another problem we need to tackle. Do you want to join our team so we can get your input there?
Happy hacking! Florian
On 19/05/16 21:25, Florian Snow wrote:
Debian has only had a major upgrade once every two years.
That's not what I mean; I mean regular updates. They should not break anything in Debian, I understand that. My point is just that we may have to use a Wordpress plugin (for example for LDAP) that may or may not play nice with all versions of Wordpress. In an ideal world, we would just not use such a plugin, but in reality, we may need it to provide the service we want to provide.
In the past, the system hackers installed upgrades, and, I presume, had to check if the blog was still running. We want to reduce the workload of the admins here by having a separate team take care of anything service (e.g. blog) related. The sysadmins should be able to install an OS update without having to check the blog specifically.
One way to deal with this is to package the plugin for Debian too, I've done that for various Drupal modules. Then the packages are being used and tested by more people.
It also means you can install the packages on a test server with minimal effort and evaluate upgrades before doing the upgrade on a real server.
I would hope the system-hackers group would coordinate the date of any major OS upgrade with you.
I would hope so for a switch from Jessie to Stretch for example. For regular updates, I think no such interaction should be required. Now, the problem I was talking about is that by using a non-packaged version of Wordpress, we decouple things a bit.
The scenario with using the packaged version could look like this: A sysadmin installs updates and one of those is for Wordpress. This update breaks LDAP authentication and possibly even brings down the whole Wordpress instance. That means the blog hackers have to act immediately to fix the issue, at a time that may be inconvenient.
You could use dpkg to put a hold on the Wordpress package version, but then you wouldn't get the security updates.
The scenario with upstream Wordpress could look like this: A sysadmin installs updates and Wordpress is not affected. Wordpress tries to do an automatic update and cannot do it because of the LDAP plugin. That means the blog hackers still need to act soon, but they can start when it is more convenient and when they have more time to actually track down and fix the problem.
As packaged systems are very standard, you should be able to easily replicate the server in a test environment and do a trial run of the upgrade and test any plugins before upgrading the real server.
I agree this would be ideal. I am afraid that we don't have the manpower for such a setup, though.
A test server doesn't have to be a fully public hosted server, it could even be your own laptop or home PC with apache and wordpress packages installed
When you use packaged software you also benefit from the testing that other people do on their systems.
Speaking of introducing the new blog team, the blogs seem to be down?
Best Carsten
Den 19-05-2016 kl. 21:37 skrev Daniel Pocock:
On 19/05/16 21:25, Florian Snow wrote:
Debian has only had a major upgrade once every two years.
That's not what I mean; I mean regular updates. They should not break anything in Debian, I understand that. My point is just that we may have to use a Wordpress plugin (for example for LDAP) that may or may not play nice with all versions of Wordpress. In an ideal world, we would just not use such a plugin, but in reality, we may need it to provide the service we want to provide.
In the past, the system hackers installed upgrades, and, I presume, had to check if the blog was still running. We want to reduce the workload of the admins here by having a separate team take care of anything service (e.g. blog) related. The sysadmins should be able to install an OS update without having to check the blog specifically.
One way to deal with this is to package the plugin for Debian too, I've done that for various Drupal modules. Then the packages are being used and tested by more people.
It also means you can install the packages on a test server with minimal effort and evaluate upgrades before doing the upgrade on a real server.
I would hope the system-hackers group would coordinate the date of any major OS upgrade with you.
I would hope so for a switch from Jessie to Stretch for example. For regular updates, I think no such interaction should be required. Now, the problem I was talking about is that by using a non-packaged version of Wordpress, we decouple things a bit.
The scenario with using the packaged version could look like this: A sysadmin installs updates and one of those is for Wordpress. This update breaks LDAP authentication and possibly even brings down the whole Wordpress instance. That means the blog hackers have to act immediately to fix the issue, at a time that may be inconvenient.
You could use dpkg to put a hold on the Wordpress package version, but then you wouldn't get the security updates.
The scenario with upstream Wordpress could look like this: A sysadmin installs updates and Wordpress is not affected. Wordpress tries to do an automatic update and cannot do it because of the LDAP plugin. That means the blog hackers still need to act soon, but they can start when it is more convenient and when they have more time to actually track down and fix the problem.
As packaged systems are very standard, you should be able to easily replicate the server in a test environment and do a trial run of the upgrade and test any plugins before upgrading the real server.
I agree this would be ideal. I am afraid that we don't have the manpower for such a setup, though.
A test server doesn't have to be a fully public hosted server, it could even be your own laptop or home PC with apache and wordpress packages installed
When you use packaged software you also benefit from the testing that other people do on their systems.
Discussion mailing list Discussion@fsfeurope.org https://mail.fsfeurope.org/mailman/listinfo/discussion