Looking at the thread about the new blog team, it is clear that some things are dependent on the relationship that teams have with the system-hackers group
Ignoring the current configuration of the systems, what is the ideal service that the system-hackers would like to be able to provide?
E.g.
- which OS/distribution? E.g. Debian - would more than one OS be feasible or required? (this multiplies support effort, should be avoided)
- will services run in virtual machines and which technology will be used (KVM, XCP, Openstack, ...), e.g. will the blogging service run in a VM isolated from the services operated by other teams? - benefit: each team can receive major OS updates at different times - disadvantage: extra effort for system-hackers team, needs more IP addresses
- how fast can packaged security updates (e.g. from Debian) be deployed? Will the updates be automatically deployed by cron or something, or will the team aim for a 24x7 support roster? In the latter case, are more volunteers needed in the system-hackers team?
- how will major upgrades (e.g. from Debian 8.0 to 9.0) be deployed? E.g. aim to begin upgrade 3 months after the official release, individual teams using a particular server can veto the upgrade until 7 months have passed
- which services must be supported by the system-hackers (I'd suggest backups, public web site, wiki, IP address allocation, SSL certificate creation, monitoring, Git, DNS, authentication/OpenID/LDAP, mail, mailing lists, SIP, XMPP and any services that the system-hackers may depend on in the course of their own activities)
- are the system-hackers willing to support a database such as PostgreSQL for use by other teams? Can they provide a convenient way for members of other teams to take snapshots of their databases for use in test servers?
- will the system-hackers allow members of other teams to use sudo?
Maybe there are other things too, it would be good to see the system-hackers provide a list of what you would be willing to support and a separate list of what people think they need the system-hackers to support.
Hi Daniel,
Maybe there are other things too, it would be good to see the system-hackers provide a list of what you would be willing to support and a separate list of what people think they need the system-hackers to support.
I imagine this will build over time and with experience. At the moment, we're talking about one volunteer team (blog-hackers) and two volunteer system administrators (system-hackers), and I would propose to give some room to be flexible in the arrangements here before we establish any formal procedures.
Ignoring the current configuration of the systems is not easily done as it somehow rests on the idea that our two volunteer system administrators would have the resources available to build something new, different from the current configuration. This is not the case and I worry that conducting such an excersise will be difficult on both sides: it will raise the expectations from the other volunteer teams, and it will be frustrating for the system administrators who may not be able to deliver.
I would leave each team some time to work now before developing things further. So as to the current state of things:
- which OS/distribution? E.g. Debian
Debian 8
- will services run in virtual machines
Services run in isolated VMs on a Proxmox cluster. There is no lack of IPv4 addresses to support this at the moment, but all new VMs have IPv6 connectivity as well.
- how fast can packaged security updates (e.g. from Debian) be deployed?
This is a best effort deployment. The individual teams can deploy security packages quickly if they would like to, but otherwise it will be anything from 1 hour to 1 month, completely dependant upon the severity of the security update and the available time.
In the latter case, are more volunteers needed in the system-hackers team?
As you may imagine, we do indeed need more volunteer also in the system hackers team. I'm not "recruiting" for this at the moment though, as giving someone this access also means giving them access to a lot of (potentially) confidential and private information. The way I see this is that the service level volunteer teams will be the future recruitment ground for system-hackers.
- how will major upgrades (e.g. from Debian 8.0 to 9.0) be deployed?
Again, this is best effort. Typically, at the moment, we do not do any major upgrades unless changing the service at the same time.
- which services must be supported by the system-hackers (I'd suggest
backups, public web site, wiki, IP address allocation, SSL certificate creation, monitoring, Git, DNS, authentication/OpenID/LDAP, mail, mailing lists, SIP, XMPP and any services that the system-hackers may depend on in the course of their own activities)
At the moment, all services which are not explicitly delegated to any other volunteer team.
- are the system-hackers willing to support a database such as
PostgreSQL for use by other teams? Can they provide a convenient way for members of other teams to take snapshots of their databases for use in test servers?
No.
- will the system-hackers allow members of other teams to use sudo?
Yes.
On 19/05/16 11:33, Jonas Oberg wrote:
Hi Daniel,
Maybe there are other things too, it would be good to see the system-hackers provide a list of what you would be willing to support and a separate list of what people think they need the system-hackers to support.
I imagine this will build over time and with experience. At the moment, we're talking about one volunteer team (blog-hackers) and two volunteer system administrators (system-hackers), and I would propose to give some room to be flexible in the arrangements here before we establish any formal procedures.
I'm not suggesting this would automatically become a formal agreement. Nonetheless, as the system-hackers are volunteers, it is actually really useful for them to specify just what they are volunteering to provide and where the limits are.
Ignoring the current configuration of the systems is not easily done as it somehow rests on the idea that our two volunteer system administrators would have the resources available to build something new, different from the current configuration. This is not the case and I worry that conducting such an excersise will be difficult on both sides: it will raise the expectations from the other volunteer teams, and it will be frustrating for the system administrators who may not be able to deliver.
I would leave each team some time to work now before developing things further. So as to the current state of things:
If there is a gap between current configuration and desired configuration and if there is long-term benefit to FSFE, additional volunteers may step forward to help the system-hackers team migrate/adapt the systems. I would certainly volunteer to help on a temporary basis if I felt the final solution would be easier for system-hackers to support.
- which OS/distribution? E.g. Debian
Debian 8
- will services run in virtual machines
Services run in isolated VMs on a Proxmox cluster. There is no lack of IPv4 addresses to support this at the moment, but all new VMs have IPv6 connectivity as well.
- how fast can packaged security updates (e.g. from Debian) be deployed?
This is a best effort deployment. The individual teams can deploy security packages quickly if they would like to, but otherwise it will be anything from 1 hour to 1 month, completely dependant upon the severity of the security update and the available time.
Security is about the weakest link in the chain. If some updates are not deployed for a month, that can let everybody else down.
In the latter case, are more volunteers needed in the system-hackers team?
As you may imagine, we do indeed need more volunteer also in the system hackers team. I'm not "recruiting" for this at the moment though, as giving someone this access also means giving them access to a lot of (potentially) confidential and private information. The way I see this is that the service level volunteer teams will be the future recruitment ground for system-hackers.
That is understandable. Even so, as more people volunteer, it will become even more important for everything to be standardized.
- how will major upgrades (e.g. from Debian 8.0 to 9.0) be deployed?
Again, this is best effort. Typically, at the moment, we do not do any major upgrades unless changing the service at the same time.
Not doing the major upgrade within 1 year of the release means you may stop getting security updates. For any public-facing service that should be a major concern. That is why I suggested putting some targets in place, although the exact upgrade dates could be varied from one service to the next, especially if they are isolated by virtualization.
- which services must be supported by the system-hackers (I'd suggest
backups, public web site, wiki, IP address allocation, SSL certificate creation, monitoring, Git, DNS, authentication/OpenID/LDAP, mail, mailing lists, SIP, XMPP and any services that the system-hackers may depend on in the course of their own activities)
At the moment, all services which are not explicitly delegated to any other volunteer team.
- are the system-hackers willing to support a database such as
PostgreSQL for use by other teams? Can they provide a convenient way for members of other teams to take snapshots of their databases for use in test servers?
No.
- will the system-hackers allow members of other teams to use sudo?
Yes.
Maybe some of this could go into a wiki page about developing a service agreement? It can have some disclaimer at the top to say it is work in progress.
Regards,
Daniel
On Thu, May 19, 2016 at 11:33:26AM +0200, Jonas Oberg wrote: ...
- how will major upgrades (e.g. from Debian 8.0 to 9.0) be deployed?
Again, this is best effort. Typically, at the moment, we do not do any major upgrades unless changing the service at the same time.
that is not 100% true...i have done the upgrade from debian 7 to 8 on some of the new systems (the first ones beein built)... for lot's of services this should be possible without a major issue if we can prevent of having lots of custom patches etc in in.
some services are more problematic then others of course...
the more interconnectiopn there are the more problematic upgrades are of course.
regards, albert
On 19/05/16 15:28, Albert Dengg wrote:
On Thu, May 19, 2016 at 11:33:26AM +0200, Jonas Oberg wrote: ...
- how will major upgrades (e.g. from Debian 8.0 to 9.0) be deployed?
Again, this is best effort. Typically, at the moment, we do not do any major upgrades unless changing the service at the same time.
that is not 100% true...i have done the upgrade from debian 7 to 8 on some of the new systems (the first ones beein built)... for lot's of services this should be possible without a major issue if we can prevent of having lots of custom patches etc in in.
some services are more problematic then others of course...
the more interconnectiopn there are the more problematic upgrades are of course.
That is one case where virtualization sometimes helps. If you have 10 services on the same host it can be impossible to upgrade without 2 days of downtime but if you have 10 different VMs then you can upgrade each one on a different day.
Hi Daniel,
Thank you for all those suggestions. I think it is a bit early to set up such an agreement. We just started the first team and we will need to figure out how to best work between the system hackers and blog hackers.
Since we don't have big requirements, the blogs are a nice testing ground for this cooperation. Right now, my suggested guideline would be: Anything OS related is the system hackers' responsibility and anything application/service related the blog hackers' responsibility. That line may become a bit more blurred in the future and then we need to decide how to continue from there. Right now, the differentiation is pretty clear though, especially if we use Wordpress from upstream.
I am a big fan of planning things ahead of time, but at this time, I wouldn't know what to plan. We have a running setup that needs to be migrated. The result will probably not be the ideal solution yet, but a step to make the blogs reliable again. For the time being, the blog hackers team need to find out the necessary steps for the migration. Generally, we are talking about exporting the database and the files, and then testing upgrade paths on the new system. There may be additional in-between steps that we will need to determine first. For example, this morning, the MySQL dump did not run cleanly and that means the database needs to be fixed first. There are probably other things we need to figure out along the way.
After that, we will have a better overview of what we have and what we think we should have. This will enable us to make good decisions about the future of the platform in regards to technology. That may be the time to set up the kind of service agreement you suggested.
For the time being, I am happy to just contact Albert quickly to say "Hey, would you please have a look at this.". That is also a more efficient use of ressources with limited man-power right now.
Happy hacking! Florian
On 19/05/16 21:26, Florian Snow wrote:
Hi Daniel,
Thank you for all those suggestions. I think it is a bit early to set up such an agreement. We just started the first team and we will need to figure out how to best work between the system hackers and blog hackers.
I agree it is a bit early to make it a formal agreement
It is not too early to start drafting it though
Given the size of the system-hackers team, it is actually really important for them to declare what they are willing and able to support, it is that simple.
How they do that may be based on some of the suggestions in my original email and/or examples from similar volunteers in other organizations (e.g. the DSA team in Debian)
Since we don't have big requirements, the blogs are a nice testing ground for this cooperation. Right now, my suggested guideline would be: Anything OS related is the system hackers' responsibility and anything application/service related the blog hackers' responsibility. That line may become a bit more blurred in the future and then we need to decide how to continue from there. Right now, the differentiation is pretty clear though, especially if we use Wordpress from upstream.
I am a big fan of planning things ahead of time, but at this time, I wouldn't know what to plan. We have a running setup that needs to be migrated. The result will probably not be the ideal solution yet, but a step to make the blogs reliable again. For the time being, the blog hackers team need to find out the necessary steps for the migration. Generally, we are talking about exporting the database and the files, and then testing upgrade paths on the new system. There may be additional in-between steps that we will need to determine first. For example, this morning, the MySQL dump did not run cleanly and that means the database needs to be fixed first. There are probably other things we need to figure out along the way.
I wasn't aware MySQL was involved. Is there any interest in moving to PostgreSQL? For example, all Debian infrastructure uses PostgreSQL now and if you aim to follow the support model for Debian infrastructure, it may be a good idea to use the same SQL back-end too.
After that, we will have a better overview of what we have and what we think we should have. This will enable us to make good decisions about the future of the platform in regards to technology. That may be the time to set up the kind of service agreement you suggested.
For the time being, I am happy to just contact Albert quickly to say "Hey, would you please have a look at this.". That is also a more efficient use of ressources with limited man-power right now.
Happy hacking! Florian _______________________________________________ Discussion mailing list Discussion@fsfeurope.org https://mail.fsfeurope.org/mailman/listinfo/discussion
Hi Daniel,
Daniel Pocock daniel@pocock.pro writes:
It is not too early to start drafting it though
For us, it still is. We need to get an overview of the current blog system first.
Given the size of the system-hackers team, it is actually really important for them to declare what they are willing and able to support, it is that simple.
We will ask them to support less than they currently support, so that is a step in the right direction. That is also all we can do for now. The blog hackers are still in the formation stage.
Is there any interest in moving to PostgreSQL?
Not at the moment, at least not for the blogs. For me, the goal is to mov away from a database alltogether and go static. But we will have to see what the team says.
Happy hacking! Florian
Il giorno gio 19 mag 2016 alle 21:59, Florian Snow floriansnow@fsfe.org ha scritto:
Not at the moment, at least not for the blogs. For me, the goal is to mov away from a database alltogether and go static. But we will have to see what the team says.
My favorite static site generator is Hugo. There's also some interest in a Hugo web-based editor (to open it to non-geeks): https://discuss.gohugo.io/t/web-based-editor/155
On 19/05/16 21:59, Florian Snow wrote:
Hi Daniel,
Daniel Pocock daniel@pocock.pro writes:
It is not too early to start drafting it though
For us, it still is. We need to get an overview of the current blog system first.
Given the size of the system-hackers team, it is actually really important for them to declare what they are willing and able to support, it is that simple.
We will ask them to support less than they currently support, so that is a step in the right direction. That is also all we can do for now. The blog hackers are still in the formation stage.
Is there any interest in moving to PostgreSQL?
Not at the moment, at least not for the blogs. For me, the goal is to mov away from a database alltogether and go static. But we will have to see what the team says.
OK, if you do end up using a database, it seems that both Debian and Fedora infrastructure are also using PostgreSQL:
https://infrastructure.fedoraproject.org/infra/docs/database.rst
Hi everyone,
Looking at the thread about the new blog team, it is clear that some things are dependent on the relationship that teams have with the system-hackers group
Ignoring the current configuration of the systems, what is the ideal service that the system-hackers would like to be able to provide?
With this as input, I now took the time to summarise some of the thoughts around this and start writing a draft service agreement for our infrastructure. I created a small team (sla-hackers) where we can discuss this further:
https://wiki.fsfe.org/Teams/SLA-Hackers
If you want to join this team, all you need to do is to join its QuickML mailing list, which you do by sending a mail to sla-hackers@q.fsfe.org with yourself in Cc [^1].
What I put together so far in terms of a draft is available here:
https://public.pad.fsfe.org/p/Zservicelevel
[^1]: http://0xcc.net/quickml/ml-usage.en.html
On Mittwoch, 24. August 2016 14:28:49 CEST Jonas Oberg wrote:
If you want to join this team, all you need to do is to join its QuickML mailing list, which you do by sending a mail to sla-hackers@q.fsfe.org with yourself in Cc [^1].
AFAIK this is won't work (and is often cause for confusion with the quickml interface).
To join the list, send a mail to sla-hackers@q.fsfe.org with Jonas (or any other member of the list) in the CC. [http://0xcc.net/quickml/ml-usage.en.html#label-4]
Johannes
Hi Johannes!
AFAIK this is won't work (and is often cause for confusion with the quickml interface).
Thanks so much for filling in with those details! I missed this subtlety completely.