Max Mehl (Code)

INWX DNS Recordmaster - Manage your DNS nameserver records via files in Git

Thu, 07 Nov 2024 00:00:00 +0000

I own and manage 30+ domains at INWX, a large and professional domain registrar. Although INWX has a somewhat decent web interface, it became a burden for me to keep an overview of each domain’s sometimes dozens of records. Especially when e.g. changing an IP address for more than one domain, it caused multiple error-prone clicks and copy/pastes that couldn’t be reverted in the worst case. This is why I created INWX DNS Recordmaster which I will shortly present here.

If you are an INWX customer, you can use this tool to manage all your DNS records in YAML files. Ideally, you will store these files in a Git repository which you can use to track changes and roll back in case of a mistake. Having one file per domain provides you a number of further advantages:

You can easily copy/paste records from other domains, e.g. for SPF, DKIM or NS records
Overall search/replace of certain values becomes much easier, e.g. of IP addresses
You can prepare larger changes offline and can synchronise once you feel it’s done

INWX DNS Recordmaster takes care of making the required changes of the live records so that it matches the local state. This is done via the INWX API, ensuring that the amount of API calls is minimal.

This even allows you to set up a pipeline that takes care of the synchronisation¹.

Wait, there is more

As written above, I already had a large stack of domains that I previously managed via the web interface. This is why some additional convenience features found their way into the tool.

You can convert all records of an existing and already configured domain at INWX into the file format. This made onboarding my 30+ domains a matter of a few minutes.
On a global or per-domain level, you can ignore certain record types. For example, if you don’t want to touch any NS records, you can configure that. By default, SOA records are ignored. You may even ignore all live records that don’t exist in your local configuration.
Of course, you can make a dry run to see which effects your configuration will have in practice.

Did I miss something to make it more productive for you? Let me know!

Install, use, contribute

You are welcome to install this tool, it’s Free and Open Source Software after all. All you need is Python installed.

One of the tool’s users is the OpenRail Association which manages some of its domains with this program and published its configuration. This is a prime example of how organisation can make the management of records transparent and easy to change at least internally, if not even externally.

While the tool is not perfect, it already is a huge gain for efficiency and stability of my IT operations, and it already proves its capabilities for other users. To reach the remaining 20% to perfection (that will take 80% of the time, as always), you are most welcome to add issues with enhancement proposals, and if possible, also pull requests.

For example, see the workflow file of the OpenRail Association. ↩︎

Seafile Mirror - Simple automatic backup of your Seafile libraries

Fri, 22 Sep 2023 00:00:00 +0000

I have been using Seafile for years to host and synchronise files on my own server. It’s fast and reliable, especially when dealing with a large number and size of files. But making reliable backups of all its files isn’t so trivial. This is because the files are stored in a layout similar to bare Git repositories, and Seafile’s headless tool, seafile-cli, is… suboptimal. So I created what started out as a wrapper for it and ended up as a full-blown tool for automatically synchronising your libraries to a backup location: Seafile Mirror.

My requirements

Of course, you could just take snapshots of the whole server, or copy the raw Seafile data files and import them into a newly created Seafile instance as a disaster recovery, but I want to be able to directly access the current state of the files whenever I need them in case of an emergency.

It was also important for me to have a snapshot, not just another real-time sync of a library. This is because I also want to have a backup in case I (or an attacker) mess up a Seafile library. A real-time sync would immediately fetch that failed state.

I also want to take a snapshot at a configurable interval. Some libraries should be synchronised more often than others. For example, my picture albums do not change as often as my miscellaneous documents, but they use at least 20 times the disk space and therefore network traffic when running a full sync.

Also, the backup service must have read-only access to the files.

A version controlled backup of the backup (i.e. the plain files) wasn’t in scope. I handle this separately by backing up my backup location, which also contains similar backups of other services and machines. For this reason, my current solution does not do incremental backups, even though this may be relevant for other use cases.

The problems

Actually, seafile-cli should have been everything you’d need to fulfill the requirements. But no. It turned out that this tool has a number of fundamental issues:

You can make the host the tool is running on a sync peer. However, it easily leads to sync errors if the user just has read-only permissions to the library.
You can also download a library but then again it may lead to strange sync errors.
It requires a running daemon which crashes irregularly during larger sync tasks or has other issues.
Download/sync intervals cannot be set manually.

The solution

seafile-mirror takes care of all these stumbling blocks:

It downloads/syncs defined libraries in customisable intervals
It de-syncs libaries immediately after they have been downloaded to avoid sync errors
You can force-re-sync a library even if its re-sync interval hasn’t reached yet
Extensive informative and error logging is provided
Of course created with automation in mind so you can run it in cronjobs or systemd triggers
And as explained, it deals with the numerous caveats of seaf-cli and Seafile in general

Full installation and usage documentation can be found in the project repository. Installation is as simple as running pip3 install seafile-mirror, and a sample configuration is provided.

In my setup, I run this application on a headless server with systemd under a separate user account. Therefore the systemd service needs to be set up first. This is also covered in the tool’s documentation. And as an Ansible power user, I also provide an Ansible role that does all the setup and configuration.

Possible next steps

The tool has been running every day since a couple of months without any issues. However, I could imagine a few more features to be helpful for more people:

Support of login tokens: Currently, only user/password auth is supported which is fine for my use-case as it’s just a read-only user. This wouldn’t be hard to fix either, seafile-cli supports it (at least in theory). (#2)
Support of encrypted libraries: Shouldn’t be a big issue, it would require passing the password to the underlying seafile-cli command. (#3)

If you have encountered problems or would like to point out the need for specific features, please feel free to contact me or comment on the Mastodon post. I’d also love to hear if you’ve become a happy user of the tool 😊.

Docker2Caddy - An automatic Reverse Proxy for Docker containers

Mon, 25 Apr 2022 00:00:00 +0000

So you have a number of Docker containers running web services which you would like to expose to the outside? Well, you probably will at least have considered a reverse proxy already. Doing this manually for one, two or even five containers may be feasible, but everything above that will be a PITA for sure. At the FSFE we ran into the same issue with our own distributed container infrastructure at and crafted a neat solution that I would like to present to you in the next few minutes.

The result is Docker2Caddy that provides a workflow in which you can spin up new containers anytime (e.g. via a CI) and the reverse proxy will just do the rest for you magically.

The assumptions

Let’s assume you want to go with reverse proxies to make your web services accessible via ports 80 and 443. There are other possibilities, and in more complex environments there may be already integrated solutions, but for this article we’ll wade in a rather simple environment spun up with docker-compose¹.

Let’s also assume you care about security and go with a rootless installation of Docker. So the daemon will run as an unprivileged user. That’s possible but much more complex than the default rootful installation². Because of this, a few other solutions will not work, we’ll check that later.

Finally, each container shall at least have one separate domain assigned to it for which you obviously want to have a valid certificate, e.g. by Let’s Encrypt.

In the examples below, we have two containers running, each running a webserver listening to port 8080. The first container shall be available via first.com, the second via second.net. The latter shall also be available via www.second.net.

The problems

In the described scenario, there are a number of problem for automating the configuration of the reverse proxy in order to direct a domain to the correct container, starting with container discovery to IPv6 routing to handling offline containers.

The reverse proxy has to be able to discover the currently running containers and ideally monitor for changes regularly so that a newly created container with a new domain is reachable within a short time without manual intervention.

Before Docker2Caddy we have used nginx-proxy combined with acme-companion (formerly known as docker-letsencrypt-nginx-proxy-companion). These are Docker containers that query all containers connected to the bridge Docker network. For this to work, the containers have to run with environment variables indicating the desired domains and local ports that shall be proxied.

In a rootless Docker setup this finally reaches its limits although discovery still works. But already before that we did not like the fact that we had to connect containers to the bridge network upon creation and therefore lost a bit more isolation (which is dubious in Docker anyway).

Now, with rootless, IPv6 was the turning point. Even in rootful Docker setups, IPv6 – a 20+ years old, well defined standard protocol – is a pain in the butt. But with rootless, the FSFE System Hackers team did not manage to get IPv6 working in containers to the degree that we needed. While IPv6 traffic reached the nginx-proxy, it was then treated as IPv4 traffic with the internal Docker IP address. That bits you ultimately if you limit requests based on IP addresses, e.g. for signups or payments. All traffic via IPv6 will be treated as the same internal IPv4 address, therefore triggering the limits regularly.

The easiest solution therefore is to use a reverse proxy running on the host system, not as a Docker container with its severe limitations. While the first intuition lead us to nginx, we decided to go with Caddy. The main advantages we saw are that a virtual host in Caddy is very simple to configure and that TLS certificates are generated and maintained automatically without extra dependencies like certbot.

In this setup, containers would need to open their webserver port to the host. This “public” port has to be unique per host, but the internal port can stay the same, e.g. port 1234 could be mapped to port 8080 inside the container. In Caddy you would then configure the domain first.org to forward to localhost:1234. A more or less identical second example container could then expose the port 5678 to the host, again listen on 8080 internally, and Caddy would redirect second.net and www.second.net to localhost:5678.

But how does Caddy know about the currently running containers and the ports via which they want to receive traffic? And how can we handle containers that are unavailable, for instance because they crashed or have been deleted for good? Docker2Caddy to the rescue!

The solution

I already concluded that Caddy is a suitable reverse proxy for the outlined use case. But in order to be care-free, the configuration has to be generated automatically. For this to work, I wrote a rather simple Python application called Docker2Caddy that is kept running in the background via a systemd service and writes proper logs that are also rotated nicely.

This is how it works internally: it queries (in a configurable interval) the Docker daemon for running containers. For each container it looks for specific labels (that are also configurable), by default proxy.host, proxy.host_alias and proxy.port. If one or multiple containers are found – in our case two – one Caddy configuration file per container is created. This is based on a freely configurable Jinja2 template. If the configuration changed, e.g. by a new host, Caddy will be reloaded and will create a TLS certificate if needed.

But what happens if a container is unavailable? In Docker2Caddy you can configure a grace period. Until this is reached, the Caddy configuration for the container in question is not removed but could forward to a local or remote error page. Only afterwards, the configuration is removed, and Caddy reloaded subsequently.

So, what makes Docker2Caddy special? I am biased but see a number of points:

Simplicity: fundamentally it’s a 188 pure lines of code Python script.
Configurability: albeit it’s simplicity, it’s easy to configure for various needs thanks to the templates and the support for rootless Docker setups.
Adaptability: it should be rather simple to make Docker2Caddy also work for Podman, or even use different reverse proxies. Feel free to extend it before I’ll do it myself someday ;)
Performance: while I did not perform before/after benchmarks, Caddy is blazingly fast and will surely perform better on the host than in a limited Docker container.

If you’re facing the same challenges in your setup, please feel free to try it out. Installation is quite simple and there’s even a minimal Ansible playbook. If you have feedback, I appreciate reading it via comments on Mastodon (see below), email, or, if you have an FSFE account, as a new issue or patch at the main repo.

This is how a very minimal Docker service in the FSFE infrastructure looks like. For Docker2Caddy, only the docker-compose.yml file with its labels is relevant. ↩︎
If you’re interested in setting this up via Ansible, I can recommend the ansible-docker-rootless role which we integrated in our full-blown playbook for the container servers. ↩︎

The power of git-sed

Tue, 28 Jul 2020 00:00:00 +0000

In the recent weeks and months, the FSFE Web Team has been doing some heavy work on the FSFE website. We moved and replaced thousands of files and their respective links to improve the structure of a historically grown website (19+ years, 23243 files, almost 39k commits). But how to do that most efficiently in a version controlled system like Git?

In our scenarios, the steps executed often looked like the following:

Move/rename a directory full of XML files representing website pages
Find all links that pointed to this directory, and change them
Create a rewrite rule

For the first step, using the included git mv is perfectly fine.

For the second, we would usually need a combination of grep and sed, e.g.:

grep -lr "/old/page.html" | xargs sed 's;/old/page.html;/new/page.html;g'

This has a few major flaws:

In a Git repository, this also greps inside the .git directory where we do not want to edit files directly
The grep is slow in huge repositories
The searched old link has to be mentioned two times, so hard for semi-manual replacement of a large number of links
Depending on the Regex complexity we need, the command becomes long, and we need to take care of using the correct flags for grep and sed.

git-sed to the rescue

After some research, I found git-sed, basically a Bash file in the git-extras project. With some modifications (pull request pending) it’s the perfect tool for mass search and replacement.

It solves all of the above problems:

It uses git grep that ignores the .git/ directory, and is much faster because it uses git’s index.
The command is much shorter and easier to understand and write
Flags are easy to add, and this only has to be done once per command

Install

You can just install the git-extras package which also contains a few other scripts.

I opted for using it standalone, so downloaded the shell file, put it in a directory which is in my $PATH, and removed one dependency on a script which is only available in git-extras (see my aforementioned PR). So for instance, you could copy git-sed.sh in /usr/local/bin/ and make it executable. To enable calling it via git sed, put in your ~/.gitconfig:

[alias]
  sed = !sh git-sed.sh

Usage

After installing git-sed, the command above would become:

git sed -f g "/old/page.html" "/new/page.html"

My modifications also allow people to use extended Regex, so things like reference captures, so I hope these will be merged soon. With this, some more advanced replacements are possible:

# Use reference capture (save absolute link as \1)
git sed -f g "http://fsfe.org(/.*?\.html)" "https://fsfe.org\1"

# Optional tokens (.html is optional here)
git sed -f g "/old/page(\.html)?" "/new/page.html"

And if you would like to limit git-sed to a certain directory, e.g. news/, that’s also no big deal:

git sed -f g "oldstring" "newstring" -- news/

You may have notived the -f flag with the g argument. People used to sed know that g replaces all appearances of the searched pattern in a file, not only the first one. You could also make it gi if you want a case-insensitive search and replace.

Conclusion

As you can see, using git-sed is really a time and nerve saver when doing mass changes on your repositories. Of course, there is also room for improvement. For instance, it could be useful to use the Perl Regex library (PCRE) for the grep and sed to also allow for look-aheads or look-behinds. I encourage you to try git-sed and make suggestions to upstream directly to improve this handy tool.

I love the hidden champions

Fri, 14 Feb 2020 00:00:00 +0000

A few days ago I’ve sent an announcement email for today’s I Love Free Software Day to a large bunch of people. Most of the remarkably many replies have been positive and a pure joy to read, but some were a bit sceptical and critical. These came from Free Software contributors who are maintaining and helping projects that they think nobody knows and sees – not because these software projects are unused, but because they are small, a building block for other, more popular applications.

When we ask people to participate in #ilovefs (this year for the 10th time in a row!) by expressing their gratitude to contributors of their favourite Free Software projects, many think about the applications they often use and come up with obvious ones like Mozilla’s Firefox and Thunderbird, LibreOffice, their Linux-based distribution, or CMSs like WordPress and Drupal. Not that I think this is not deserved, but what about the projects that actually form the foundations for these popular suites?

I researched a bit on my own system (based on Arch Linux) and checked on how many packages some of the aforementioned applications depend (including dependencies of their dependencies)¹:

Firefox: 221
Thunderbird: 179
LibreOffice: 185
GIMP: 166
Inkscape: 164

Phew! Looking through the list of dependencies, there a dozens of programmes and libraries that I couldn’t even imagine what they could be about. But they make a big application, be it Firefox, Thunderbird or GIMP, actually possible. Isn’t it a bit unfair that we often don’t see these small (or sometimes huge) projects and the people who take care of it?²

I decided to change that, at least for one day! I’ve analysed which packages are most used as dependencies of other packages (similar for Debian/Ubuntu ³):

for p in $(pacman -Q | cut -d" " -f1); do
  echo "$(pactree -r -l $p | tail -n+2 | sort | uniq | wc -l)–$p–$(pacman -Qi $p | grep "^Description" | grep -oP '(?<=: ).*')"
done | column -t -s'–' | sort -nr

Output:

1621  iana-etc                   /etc/protocols and /etc/services provided by IANA
1620  tzdata                     Sources for time zone and daylight saving time data
1620  linux-api-headers          Kernel headers sanitized for use in userspace
1620  filesystem                 Base Arch Linux files
1619  glibc                      GNU C Library
1349  gcc-libs                   Runtime libraries shipped by GCC
1287  ncurses                    System V Release 4.0 curses emulation library
1267  readline                   GNU readline library
1261  bash                       The GNU Bourne Again shell
...

As you might expect, on the very top I found a lot of GNU and Linux sub-projects, some widely known (like bash), some which I as a more-user-than-developer never heard of before (like libffi). This alone has been an interesting journey during which I learnt a lot about projects and their maintainers which play a crucial role on my laptop.

In the end, I decided to express my thanks today to the following projects and people:

The development team behind acl/attr which controls access permissions
The four initial creators of argon2, Jean-Philippe, Samuel, Dmitry and Daniel, for their password hasing function
Jan Dittberner (who also is a FSFE supporter!) and Nathan Neulinger, developers of CrackLib which checks and enforces strong passwords
Reuben Thomas and Dom Lachowicz for their enchant project, a wrapper for various spell checking engines
Maintainers of glibc and gcc, important tools for the C library and compiler
The HarfBuzz team which can shape glyphs from Unicode texts
The libmnl/netfilter people, who provide tools for network-related operations
The contributors of libxml2 for their library and tools that are crucial for the FSFE website
Martin Mitáš who more or less alone maintains md4c, a Markdown parser
Thomas Dickey who maintains ncurses which provides a text-based interface for the command line
Chet Ramey as representative of readline, a programme for interactive user input
And last but not least Lasse Collin who maintains xz, a compression tool

But of course, that’s only a small fraction of the many interesting Free Software components that enable my daily work. However, if we all do the same and think about the hidden champions – not only during #ILoveFs day but beyond – we can make the humans behind it enjoy their invaluable contributions a bit more.

Happy I Love Free Software Day everyone! ❤

pactree -l firefox | sort | uniq ↩︎
During the writing of this blog post I remembered Matthias hugging Peter Stuge for #ilovefs 2013 who also contributes to widely used Free Software projects. ↩︎

If you want to try the same with apt (with another separator):

for p in $(dpkg --get-selections | cut -f1 | cut -d":" -f1); do
  echo "$(apt-cache rdepends $p | tr -d '|' | tail -n+3 | sort | uniq | wc -l)*$p*$(apt-cache show $p | grep -m 1 "^Description:" | grep -oP '(?<=: ).*')"
done | column -t -s'*' | sort -nr

↩︎

Build FSFE websites locally

Sun, 13 Nov 2016 23:00:44 +0000

Note: This guide is also available in FSFE’s wiki now, and it will be the only version maintained. So please head over to the wiki if you’re planning to follow this guide.

Those who create, edit, and translate FSFE websites already know that the source files are XHTML files which are build with a XSLT processor, including a lot of custom stuff. One of the huge advantages from that is that we don’t have to rely on dynamic website processors and databases, on the other hand there are a few drawbacks as well: websites need a few minutes to be generated by the central build system, and it’s quite easy to mess up with the XML syntax.

Now if an editor wants to create or edit a page, she needs to wait a few minutes until the build system has finished everytime she wants to test how the website looks like. So in this guide I will show how to build single websites on your own computer in a fraction of the FSFE’s system build time, so you’ll only need to commit your changes as soon as the file looks as you want it. All you need is a bit hard disk space and around one hour time to set up everything.

The whole idea is based on what FSFE’s webmaster Paul Hänsch has coded and written. On his blog he explains the new build script. He explains how to build files locally, too. However, this guide aims to make it a bit easier and more verbose.

Before we’re getting started, let me shortly explain the concept of what we’ll be doing. Basically, we’ll have three directories: trunk, status, and fsfe.org. Most likely you already have trunk, it’s a clone of the FSFE’s main SVN repository, and the source of all operations. All those files in there have to be compiled to generate the final HTML files we can browse. The location of these finished files will be fsfe.org. status, the third directory, contains error messages and temporary files.

After we (1) created these directories, partly by downloading a repository with some useful scripts and configuration files, we’ll (2) build the whole FSFE website on our own computer. In the next step, we’ll (3) set up a local webserver so you can actually browse these files. And lastly we’ll (4) set up a small script which you can use to quickly build single XHTML files. Last but not least I’ll give some real-world examples.

1. Clone helper repository

Firstly, clone a git repository which will give you most needed files and directories for the further operations. It has been created by me and contains configuration files and the script that will make building of single files easier. Of course, you can also do everything manually.

In general, this is the directory structure I propose. In the following I’ll stick to this scheme. Please adapt all changes if your folder tree looks differently.

trunk (~700 MB):      ~/subversion/fsfe/fsfe-web/trunk/
status (~150 MB):     ~/subversion/fsfe/local-build/status/
fsfe.org (~1000 MB):  ~/subversion/fsfe/local-build/fsfe.org/

(For those not so familiar with the GNU/Linux terminal: ~ is the short version of your home directory, so for example /home/user. ~/subversion is the same as /home/USER/subversion, given that your username is USER)

To continue, you have to have git installed on your computer (sudo apt-get install git). Then, please execute via terminal following command. It will copy the files from my git repository to your computer and already contains the folders status and fsfe.org.

git clone https://src.mehl.mx/mxmehl/fsfe-local-build.git ~/subversion/fsfe/local-build

Now we take care of trunk. In case you already have a copy of trunk on your computer, you can use this location, but please do a svn up beforehand and be sure that the output of svn status is empty (so no new or modified files on your side). If you don’t have trunk yet, download the repository to the proposed location:

svn --username $YourFSFEUsername co https://svn.fsfe.org/fsfe-web/trunk ~/subversion/fsfe/fsfe-web/trunk

2. Build full website

Now we have to build the whole FSFE website locally. This will take a longer time but we’ll only have to do it once. Later, you’ll just build single files and not >14000 as we do now.

But first, we have to install a few applications which are needed by the build script (Warning: it’s possible your system lacks some other required applications which were already installed on mine. If you encounter any command not found errors, please report them in the comments or by mail). So let’s install them via the terminal:

sudo apt-get install make libxslt

Note: libxslt may have a different name in your distribution, e.g. libxslt1.1 or libxslt2.

Now we can start building.The full website build can be started with

~/subversion/fsfe/fsfe-web/trunk/build/build_main.sh --statusdir ~/subversion/fsfe/local-build/status/ build_into ~/subversion/fsfe/local-build/fsfe.org/

See? We use the build routine from trunk to launch building trunk. All status messages are written to status, and the final website will reside in fsfe.org. Mind differing directory names if you have another structure than I do. This process will take a long time, depending on your CPU power. Don’t be afraid of strange messages and massive walls of text ;-)

After the long process has finished, navigate to the trunk directory and execute svn status. You may see a few files which are new:

max@bistromath ~/s/f/f/trunk> svn status
?       about/printable/archive/printable.en.xml
?       d_day.en.xml
?       d_month.en.xml
?       d_year.en.xml
?       localmenuinfo.en.xml
[...]

These are leftover from the full website build. Because trunk is supposed to be your productive source directory where you also make commits to the FSFE SVN, let’s delete these files. You won’t need them anymore.

rm about/printable/archive/printable.en.xml d_day.en.xml d_month.en.xml d_year.en.xml localmenuinfo.en.xml
rm tools/tagmaps/*.map

Afterwards, the output of svn status should be empty again. It is? Fine, let’s go on! If not, please also remove those files (and tell me which files were missing).

3. Set up local webserver

After the full build is completed, you can install a local webserver. This is necessary to actually display the locally built files in your browser. In this example, I assume you don’t already have a webserver installed, and that you’re using a Debian-based operating system. So let’s install lighttpd which is a thin and fast webserver, plus gamin which lighttpd needs in some setups:

sudo apt-get install lighttpd gamin

To make Lighttpd running properly we need a configuration file. This has to point the webserver to show files in the fsfe.org directory. You already downloaded my recommended config file (lighttpd-fsfe.conf.sample) by cloning the git repository. But you’ll have to modify the path accordingly and rename it. So rename the file to lighttpd-fsfe.conf, open it and change following line to match the actual and absolute path of the fsfe.org directory (~ does not work here):

server.document-root = "/home/USER/subversion/fsfe/local-build/fsfe.org"

Now you can test whether the webserver is correctly configured. To start a temporary webserver process, execute the next command in the terminal:

lighttpd -Df ~/subversion/fsfe/local-build/lighttpd-fsfe.conf

Until you press Ctrl+C, you should be able to open your local FSFE website in any browser using the URL http://localhost:5080. For example, open the URL http://localhost:5080/contribute/contribute.en.html in your browser. You should see basically the same website as the original fsfe.org website. If not, double-check the paths, if the lighttpd process is still running, or if the full website build is already finished.

4. Single page build script

Until now, you didn’t see much more than you can see on the original website. But in this step, we’ll configure and start using a Bash script (fsfe-preview.sh) I’ve written to make a preview of a locally edited XHTML file as comfortable as possible. You already downloaded it by cloning the repository.

First, rename and edit the script’s configuration file config.cfg.sample. Rename it to config.cfg and open it. The file contains all paths we already used here, so please adapt them to your structure if necessary. Normally, it should be sufficient to modify the values for LOC_trunk (trunk directory) and LOC_out (fsfe.org directory), the rest can be left with the default values.

Another feature of the fsfe-preview is to automatically check the XML syntax of the files. For this, libxml2-utils has to be installed which contains xmllint. Please execute:

sudo apt-get install libxml2-utils

Now let’s make the script easy to access via the terminal for future usage. For this, we’ll create a short link to the script from one of the binary path directories. Type in the terminal:

sudo ln -s ~/subversion/fsfe/local-build/fsfe-preview.sh /usr/bin/fsfe-preview

From this moment on, you should be able to call fsfe-preview from anywhere in your terminal. Let’s make a test run. Modify the XHTML source file contribute/contribute.en.xhtml and edit some obvious text or alter the title. Now do:

As output, you should see something like:

[INFO] Using file /home/max/subversion/fsfe/fsfe-web/trunk/contribute/contribute.en.xhtml as source...
[INFO] XHTML file detected. Going to build into /home/max/subversion/fsfe/local-build/fsfe.org/contribute/contribute.en.html ...
[INFO] Starting webserver

[SUCCESS] Finished. File can be viewed at http://localhost:5080/contribute/contribute.en.html

Now open the mentioned URL http://localhost:5080/contribute/contribute.en.html and take a look whether your changes had an effect.

Recommended workflows

In this section I’ll present a few of the cases you might face and how to solve them with the script. I presume you have your terminal opened in the trunk directory.

Preview a single file

To preview a single file before uploading it, just edit it locally. The file has to be located in the trunk directory, so I suggest to only use one SVN trunk on your computer. It makes almost no sense to store your edited files in different folders. To preview it, just give the path to the edited file as argument for fsfe-preview, just as we did in the preceding step:

fsfe-preview activities/radiodirective/statement.en.xhtml

The script detects whether the file has to be built with the XSLT processor (.xhtml files), or if it just can be copied to the website without any modification (e.g. images).

Copy many files at once

Beware that all files you added in your session have to be processed with the script. For example, if you create a report with many images included and want to preview it, you will have to copy all these images to the output directory as well, and not only the XHTML file. For this, there is the -copy argument. This circumvents the whole XSLT build process and just plainly copies the given files (or folders). In this example, the workflow could look like the following: The first line copies some images, the second builds the corresponding XHTML file which makes use of these images:

fsfe-preview --copy news/2016/graphics/report1.png news/2016/graphics/report2.jpg
fsfe-preview news/2016/news-20161231-01.en.xhtml

Syntax check

In general, it’s good to check the XHTML syntax before editing and commiting files to the SVN. The script fsfe-preview already contains these checks but it’s good to be able to use it anyway. If you didn’t already do it before, install libxml2-utils on your computer. It contains xmllint, a syntax checker for XML files. You can use it like this:

xmllint --noout work.en.xhtml

If there’s no output (–noout), the file has a correct syntax and you’re ready to continue. But you may also see something like

work.en.xhtml:55: parser error : Opening and ending tag mismatch: p line 41 and li
      </li>
           ^

In this case, this means that the <p> tag starting in line 41 isn’t closed properly.

Drawbacks

The presented process and script has a few drawbacks. For example you aren’t able to preview certain very dynamic pages or parts of pages, or those depending on CGI scripts. In most cases you’ll never encounter these, but if you’re getting active with the FSFE’s webmaster team it may happen that you’ll have to fall back on the standard central build system.

Any other issues? Feel free to report them as they will help to improve FSFE’s editors to work more efficiently :-)

Changelog

29 November 2016: Jonas has pointed out a few bugs and issues with a different GNU/Linux distribution. Should be resolved.

splitDL – Downloading huge files from slow and unstable internet connections

Fri, 26 Jun 2015 15:59:03 +0000

Imagine you want install GNU/Linux but your bandwidth won’t let you…

tl;dr: I wrote a rather small Bash script which splits huge files into several smaller ones and downloads them. To ensure the integrity, every small files is being checked for its hashsum and file size.

That’s the problem I was facing in the past days. In the school I’m working at (Moshi Institute of Technology, MIT) I set up a GNU/Linux server to provide services like file sharing, website design (on local servers to avoid the slow internet) and central backups. The ongoing plan is the setup of 5-10 (and later more) new computers with a GNU/Linux OS in contrast to the ancient and non-free WindowsXP installations – project „Linux Classroom“ is officially born.

But to install an operating system on a computer you need an installation medium. In the school a lot of (dubious) WindowsXP installation CD-ROMs are flying around but no current GNU/Linux. In the first world you would just download an .iso file and ~10 minutes later you could start installing it on your computer.

But not here in Tanzania. With download rates of average 10kb/s it needs a hell of a time to download only one image file (not to mention the costs for the internet usage, ~1-3$ per 1GB). And that’s not all: Periodical power cuts cancel ongoing downloads abruptly. Of course you can restart a download but the large file may be already damaged and you loose even more time.

My solution – splitDL

To circumvent this drawback I coded a rather small Bash program called splitDL. With this helper script, one is able to split a huge file into smaller pieces. If during the download the power cuts off and damages the file, one just has to re-download this single small file instead of the huge complete file. To detect whether a small file is unharmed the script creates hashsums of the original huge and the several small files. The script also supports continuation of the download thanks to the great default built-in application wget.

You might now think „BitTorrent (or any other program) is also able to do the same, if not more!“. Yes, but this requires a) the installation of another program and b) a download source which supports this protocol. On the contrary splitDL can handle every HTTP, HTTPS or FTP download.

The downside in the current state is that splitDL requires shell access to the server where the file is saved to be able to split the file and create the necessary hashsums. So in my current situation I use my own virtual server in Germany on which I download the wanted file with high-speed and then use splitDL to prepare the file for the slow download from my server to the Tanzanian school.

The project is of course still in an ongoing phase and only tested in my own environment. Please feel free to have a look at it and download it via my Git instance. I’m always looking forward to feedback. The application is licensed under GPLv3 or later.

Some examples

Server-side

Split the file debian.iso into smaller parts with the default options (MD5 hashsum, 10MB size)

split-dl.sh -m server -f debian.iso

Split the file but use the SHA1 hashsum and split the file into pieces with 50MB.

split-dl.sh -m server -f debian.iso -c sha1sum -s 50M

After one of the commands, a new folder called dl-debian.iso/ will be created. There, the splitted files and a document containing the hashsums and file sizes are located. You just have to move the folder to a web-accessible location on your server.

Client-side

Download the splitted files with the default options.

split-dl.sh -m client -f http://server.tld/dl-debian.iso/

Download the splitted files but use SHA1 hashsum (has to be the same than what was used on the creation process) and override the wget options (default: -nv –show-progress).

split-dl.sh -m client -f http://server.tld/dl-debian.iso/ -c sha1sum -w --limit-date=100k

Current bugs/drawbacks

Currently only single files are possible to split. This will be fixed soon.
Currently the script only works with files in the current directory. This is also only a matter of some lines of code.

I love Taskwarrior, therefore I love Free Software

Sat, 14 Feb 2015 12:05:42 +0000

“It’s Valentine’s day and you’re writing a blog post? Are you nuts?” you might ask. Well, but it’s not only Valentine’s day but also I love Free Software day. This day is proclaimed every year on February 14 by the Free Software Foundation Europe to thank all developers and contributors of Free Software (software you can use for any purpose, which source code you or others can analyze, which can be modified and distributed).

As last year with ZNC, I want to say thank you to a specific project which easies my daily life. As you might know by other blog posts here, organisation of tasks, mails and almost everything else is a very important issue for me. So this year I want to write some lines about Taskwarrior, taskd and Mirakel which enable me to take some free time without thinking of task which I could possibly forget to accomplish later on.

My head is full of ideas and mental To-Do lists and so I’m in need of a handy tool which allows me to write down and organise items at any place and time: At my desk, in bus or train, when I’m offline or abroad. And its important that I don’t have (analog and digital) bits of paper everywhere, so I need a system that syncs all task inputs and outputs. I tried a lot of tools but Taskwarrior was the best so far. It used the well-known „Getting Things Done“ concept with different priorities. Taskwarrior also supports tagging tasks, organising them in projects, due dates, postponing, making tasks dependend on others and much more. And Taskwarrior has a (modifiable) algorhythm that sorts your tasks by urgency levels, so that the most important tasks always are on the top of the list. Even now I just took a glance at what Taskwarrior is able to do!

Someone who loves Taskwarrior as much as I do

“Services and programs that organise tasks aren’t very special!” one might think. But if you prefer sorting tasks digitally, you cannot simply chose a random todo-organising service provider. Most of the tools and services on the market aren’t free and transparent. All input may no longer belong to you, all the gathered information (which is a lot if you think of it!) could be used for targeted ads or worse. You cannot modify the algorhythm to suit your needs. And what happens if the service provider goes bankrupt? All data, all project history and all pending tasks would be lost at once. So using a free (as in freedom), decentralised, maybe self-hosted service is the best idea to organise your tasks decentrally.

But one thing at a time, let’s start from the very basic. You can install Taskwarrior and almost any operating system. After the installation, taskwarrior isn’t much more than a black window with white letters in it. And even when you’re a pro-user, you won’t find much more than white or colourful text on black background – and this is a good thing! I’ve seen no graphical user interface which can handle Taskwarrior’s complexity and the users‘ needs sufficiently (but there are some, feel free to test them!). Nevertheless, it’s quite easy to use Taskwarrior from your terminal:

task add "This is my first task"          # Add your first item
task long                                 # Show all pending tasks
task add "Second VIP task!" pri:H         # Add a task with priority
task add "Third task with tag" +test      # Add a task with a tag
task add "Fourth projected task" pro:Blog # Add a task with a project
task long                                 # Show all pending tasks
task 1 done                               # Mark first task as done (ID = 1)

There are many useful and well understandable guides in the project’s documentation. Most likely you do not need every command but maybe it’s useful to read something about techniques which might help you to organise your tasks your way.

Some useful commands of Taskwarrior using some fish shell features

But Taskwarrior is only for your local computer. What’s if you want to use it when sitting in the bus and don’t want to forget a ToDo item you want to write down at the very moment? Then there’s a handy application for Android called Mirakel. Even the app itself is powerful, but it’s full potential is unleashed when combining it with Taskwarrior. For this, we need a central instance which synchronises the tasks you add or edit on your devices. The Taskwarrior project developed taskd for it which you can easily setup on a server. You can also use Mirakel’s own public taskd server (at least in the past) if you don’t own a server or don’t want to maintain this service.

So if you connect both Taskwarrior and Mirakel to the new taskd server, you can easily share all tasks among them. When marking a task done on your smartphone, it’s marked as done on your home computer some seconds or minutes later if you want to. Security is an important part of taskwarrior as well, so transport encryption is on by default. And if you want, you can also try a web interface or other handy tools and extensions for your server and client which I haven’t tested yet.

Hopefully you now know a bit more about Taskwarrior and Mirakel and the great tools they designed. Of course I do not only want to recommend some software but also use this opportunity to say a big THANK YOU to all the people behind these projects! Thank your for developing the software and making it compatible to each other. Thanks to the various contributors which are writing the important documentation, adding new languages, writing tools and bridges for other usage scenarios and thank you for reacting to bug reports. People like you make Free Software possible!

Mounting a SFTP storage in GNU/Linux

Mon, 13 Jan 2014 14:42:01 +0000

This (longer than expected) post explains how to transfer files securely between your device and an external storage. The first part may be useful for you if you only have little knowledge of terms like (S)FTP(S) and want to learn something about widely used technologies. The second part will help you to mount an external storage so you can manage all files as if they are on your local device and the third, fourth and fifth part will concentrate on easing the mounting process by the help of hostnames, Private/Public Keys and a shell script.

This guide will be very detailed and is also (and especially) suited for beginners. Maybe also some advanced users can learn something or give hints for improvements.

Update: With improving Bash skills and more time, I was able to heavily improve the script in the end. Have a look at my Git instance to download the latest version.

But let’s be honest: All in all, this post will show you again, why Free Software, GNU/Linux and Open Standards are great, easy to use and why Windows users are to be pitied.

Short excursus

(Nearly) everybody knows FTP. FTP is a protocol which enables you to transfer files between your device and a remote space. Maybe you want to present your documents or images to visitors of your homepage and simply want to upload these files on your webspace. In most cases this could be done by the use of a seperate program like FileZilla.

So far so good, but there’re several problems. Two of them:

FTP is insecure. Period.
Using an external program (and not your personal file manager) is really annoying if you want to edit the files very often. A realistic example: You have a complicated script running on your website which you’d like to edit in a graphical editor. Using an external client forces you to download the file, open it in your editor, save it and upload it again. Some FTP clients like FileZilla have the functionality to ease this pain in the a**, but trust me: after the twentieth reupload you want to toss your computer away…

Now we know why FTP is insecure. So what alternatives do we have?

There is FTPS – using the FTP protocoll in connection with SSL. Well, that sounds great because, you know, SSL is great to ensure the safety of your online banking and frightens away sniffing script kiddies. Forget that. FTPS is only securing the authentication process, the transfer of all files remains in plain text. So if you upload a sensitive document, it is fairly easy to intercept it on different levels in the uploading/downloading process. There is also the fact that FTPS is horrible to implement, not very compatible, and not very smart designed.

The best alternative in my opinion is SFTP (you are confused by all the abbreviations now? Wait for it, it gets better). SFTP uses a SSH connection to build a safe tunnel between your device and the remote storage. SSH is very smart, widely used and on most servers running a Free operating system it is preinstalled and needs no further configuration. SFTP enforces you to transfer your files fully encrypted through a secure channel.

But how to use it? You can use clients like FileZilla but do you remember the first problem I mentioned? Correct, an external editor is not what suits the needs of someone who wants to work with his files comfortably.

Mounting an external storage with SSH

This sounds more complicated than it really is. The theory behind it is very easy: You mount (link) a remote directory over SSH (a secure channel) in any desired directory on your machine. For instance, you can simply edit all files in /home/user/remote/ with your programs, but finally all changes will happen (nearly) instantly on your remote storage. This even works with watching a video file that is located on the server: Open it like a normal video file in a player like VLC and it behaves like it is locally on your PC (if your internet connection is fast enough, just test it!).

The only prerequisite: You need a server/webspace/storage with full SSH access. Unfortunately many webhosters don’t provide SSH access. If you belong to the unlucky ones, I recommend you to look for alternatives – it is worth it! (below this post, I added a very small list of webhosting providers offering SSH access)

Now we come to the technical part. For this post, following data is used. Most likely, this will look different in your case.

SSH-Server: server1.net
Username on server: client
Home directory of user on server: /home/client
Username local machine: user
Local mount directory: /home/user/remote/server1

On your GNU/Linux, please assure that following programs are installed. Some of them (ssh-askpass, zenity) are only used in step IV and V:

openssh-client, sshfs, ssh-askpass, zenity

Now create a directory in which the remote storage should be mounted in. Don’t worry about the space on your hard disk: This directory is remote and there are only temporary files stored on your local device. In this example I’ll use /home/user/remote/server1. It could also be /fsfe/lol/wtf/nsa/ if this directory would exist.

Now, start a terminal/shell to fill in following commands as a normal user.

ssh -p 22 client@server1.net

If asked for a password, fill in the password of ‚client‚ (the webhoster should have given it to you). If you are now connected to the server1.net, you are just one step away from mounting it. Close the connection or terminal, open it again and type:

sshfs -p 22 client@server1.net: /home/user/remote/server1/ -o follow_symlinks

This command connects to ‚server1.net‚ as the user ‚client‚ using the SSH protocol over the standard port 22. Then it mounts the home directory of ‚client‚ on the local directory ‚/home/user/remote/server1‚ which we created beforehand. Additionally we added the option ‚follow_symlinks‚ so that links of the server work on our local machine as well. If you have a look at /home/user/remote/server1, you should see the exact same content than in the home directory on your server or if you connect via FTP/SFTP. Congratulation! Now play around with it, try to edit, upload and download files.

To unmount (speak: disconnect) the directory, type

fusermount -u /home/user/remote/server1

Now you mount the home directory of ‚client‘. If you want to mount another directory on the server (e.g. /home/client/exchange), use this modified command:

sshfs -p 22 client@server1.net:/home/client/exchange/ /home/user/remote/server1/ -o follow_symlinks

In the next step, we will make the connection and mounting more comfortable, even if you are handling with more than one server.

Using hostnames

Now that you know how (and that) the system works, we will make it easier. Of course it is quite annoying to type in the whole server-address and port each time. Instead of sshfs -p 22 client@server1.net[…], you can simply type sshfs server1. How? Just open the SSH configuration file /home/user/.ssh/config with you desired text editor and add:

Host server1
  HostName server1.net
  Port 22
  User client

Save it and try to use this shortcut. It also works for normal SSH connection like ‚ssh server1‚. For the Host variable, you could use any name you can remember easily, for instance privateserver.

Using Public and Private Keys

Pretty smooth, isn’t it? Well, it gets even better! You have to admit that typing a password each time you connect is quite harrassing. This gets even worse if you are used to connect to many servers. On the one hand, you could use password managing tools… or Public/Private key authentication! For those who don’t know much about it, let me say: This sounds terribly complicated, but you don’t have to understand it completely.

For this guide, you only have to know: There are two keys (files). The one is private, you should never give it to anyone! It’s like a key to your safe, if someone achieves it, he can get everything that is secured in this safe. But the other file is public, you can give it to everyone if you want so. Now it gets a bit weird, so I try to make it as simple and abstract as possible.

Let’s say, the public key is a chest that no one except the owner can open. Inside this chest, there is a supersecret word that no one except the owner knows. You put an exact copy of this chest on your server and each time you try to connect, you say „Hey, I’m the legitimate user of this server. Give me my chest, I will open it with my secret key and tell you, what the supersecret word is!“. The server will give you the chest (public key), you open it with your private key and tell the server the supersecret word. If you were right, the server lets you in.

Note: This example is so ridiculously abstract that I had to laugh while writing it down. Every security expert, IT guy and cryptography scientist would knock me down for this but I hope it made you understand the principle of this very smart method. And if you ask yourself now „Why does the server know the supersecret word? Isn’t this insecure?“ or „Isn’t is insecure to send such a sensitive code word over the internet?“: You’re absolutely right! But be assured, that the public-key cryptography uses some awesome tricks to avoid security leaks like these and others. If you want to, read about it!

It is quite easy to make this system happen. Again we need a terminal to generate the two keys:

cd ~/.ssh/
ssh-keygen -t dsa

This neat script asks a.) in which file you want to save the new keys. The standard is set to /home/user/.ssh/id_dsa and I recommend this. Most programs automatically search for keys in this destination. In step b.) it asks you for a password. This is quite important because if someone achieves your private key, he still needs a password to use it. After this short procedure, you have two new files in /home/user/.ssh – id_dsa and id_dsa.pub. As you can imagine, id_dsa.pub is your public key, the other is the secret one.

Now we have to put the public key (chest) on the server we want to connect to. This could be done by a simple command:

ssh-copy-id -i /home/user/.ssh/id_dsa.pub client@server1.net

By this, you automatically connect as a legitimate user to your server (with the password used in step II) and put the public key in the /home/client/.ssh/authorized_keys. This file simply collects all public keys (chests) of users that are allowed to connect to the server. If a stranger tries to connect to your server but there is no chest his secret key belongs to, he of course cannot open the connection to your server.

Now disconnect from your server and try to connect via ssh server1. Your device should automatically search for your private key and open the connection with it. As you notice, you are still asked for a password. This is because you (hopefully) put a password on your private key (remember?). But after you typed in the password once, it should be saved for duration of your local session in a local keyring.

If this is not the case or you want to make your configuration even better, add the following lines on the bottom of your /home/user/.bashrc (or if you’re not using bash, the respective rc file of your shell):

alias ssh='ssh-add -l > /dev/null || ssh-add -t 7200; ssh'

By this, you say to your shell: „Everytime I type the command ’ssh‘, you should instead (alias) 1.) check if I already added my key to the local keyring. If not so (||), 2.) add my key to the keyring, but only keep it there for 7200 seconds (you can also keep this away or increase it). Only if you made this sure, use the command ssh to connect to my server“. Every command line program that uses ssh to connect to a server (and there are some, e.g. git, sshfs…) now uses this procedure – no matter if you close the terminal or lock the screen.

Of course, this also works with sshfs to mount your remote storages. If you remember our first line to mount the server, we have a much shorter line now without any password request:

sshfs server1: /home/user/remote/server1/ -o follow_symlinks

And these steps were quite important for the next section where we will write a shell script to make it much more easier – and even graphical!

Using a shell script

Our setting is very smooth now, but it could still be improved. If you want to connect to many servers and don’t want to use your shell every time or don’t want to remember the HOSTs you used in your .ssh/config, you’re free to modify and use this shell script:

#!/bin/bash

### VARIABLES TO BE CHANGED ###
# Preconfigured HOSTs in ~/.ssh/config that should be used
PRESSH[0]=server1
PRESSH[1]=server2
PRESSH[2]=server3

# Local directory where the remote storages should be mounted to
LOCALMOUNTDIR=/home/user/remote

### THE SCRIPT BEGINS HERE ###

# Add SSH key to local keyring if not already happened
function sshadd {
	ssh-add -l > /dev/null || ssh-add
}

# Choose preconfigured HOST to mount
function mount {
	if ! SSH=$(zenity --list \
		--height=300 \
		--text="Please choose. Cancel to unmount drives." \
		--title="Choose SSH server" \
		--column "Preconfigured SSH servers" \
		${PRESSH[*]}); then
		unmountquestion		# If you press cancel, it should ask you to unmount all drives
	fi

	# If you double click on an entry, it gives 'server1|server1' as result instead of 'server1'
	# This command cuts of everything after |
	SSH=$(echo $SSH | awk -F\| '{ print $1 }')

	# Make a local directory if not available
	if [ ! -e "$LOCALMOUNTDIR"/"$SSH" ]; then
		mkdir -p "$LOCALMOUNTDIR"/"$SSH"
	fi

	# Command to mount actually
	sshfs "$SSH": "$LOCALMOUNTDIR"/"$SSH"/ -o follow_symlinks &

	quitquestion	# one more ssh server or quit?

}

# Ask if all preconfigured SSHFS drives should be unmounted
function unmountquestion {
	zenity --question --text="Unmount all preconfigured\nSSHFS drives now?"
	if [ "$?" = "0" ]; then
		unmount		# unmount function
	else
		exit 0
	fi
}

# Procedure to unmount all preconfigured SSHFS drives and exit program afterwards
function unmount {
	for ((i = 0; i < ${#PRESSH[*]}; i++))
	do
		fusermount -u "$LOCALMOUNTDIR"/"${PRESSH[$i]}"
		echo ""${PRESSH[$i]}" unmounted."
	done
	exit 0
}

# Should another SSHFS storage be mounted?
function quitquestion {
	zenity --question \
	--text="Mount another SSHFS storage?"
	if [ "$?" = "1" ]; then
		exit 0
	fi
}

sshadd		# sshadd function

# Loop for endless mounts until stopped by unmount or unmountquestion
while :
do
	mount	# mount function
done

Save this script to some place (e.g. /home/user/sftp-mount.sh) and make it executable (chmod +x /home/user/sftp-mount.sh). For example, you are now ready put a shortcut in your panel or on your desktop to make it easily accessible. Please note that zenity, sshfs and ssh-askpass should be installed.

Please test the procedures written above on your servers and (if they have SSH access) webspaces. On my 4 webspaces and my vServer it works perfectly with my Debian Sid system. Please contact me or write in the comments if you have any problems or if some steps could be improved – nobody’s perfect :)

Appendix

I know following hosters that provide webspace with SSH access. This list is very short so please add more by writing in the comments!

uberspace.de (All plans, starting from 1,00€/mon, focussed on german residents) <– great hoster by the way!!
OVH.com (>= Business Hosting, 5,94€/mon)
HostGator.com (All plans, starting from 3,96$/mon)
All-Inkl.com (>= Premium, 9,95€/mon)
Hetzner.de (>= Level19, 19,90€/mon)