This is the sixth in a series of several posts on how to do way more than you really need to with Let’s Encrypt, certbot, and a good server. I use all of these things regularly but I’ve never taken the time to take them apart, look at how they work, and spend hours in Google trying in vain to figure out how to put them back together. It was inspired by a disturbing trend of ISP privacy violations and the shocking regulatory capture of the US Federal Communications Commission.

This post looks at several different ways to automate cert renewal. I tried to cater to everyone by including cron and systemd options. If you don’t have your server set up to send emails, you might want to do that first.

The Series so Far

  1. Overview
  2. First Steps
  3. Tuning with OpenSSL
  4. Useful Headers
  5. Generating and Testing a Cert
  6. Automating Renewals

Things that are still planned but probably not soon:

  • Updating OpenSSL
  • CSP Playground
  • Vagrant Examples (don’t hold your breath)

Code

You can view the code related to this post under the post-06-automation tag. If you’re curious, you can also check out my first draft.

Note

I’m testing out some new tooling. This will be wotw-highlighter’s shakedown run. Let me know what you think about the syntax highlighting! I’m pretty excited because (no whammies) it should work well in AMP and normally.

I wrote the majority of the Apache examples with httpd in mind, i.e. from a RHEL perspective. If you instead use apache2, most of the stuff should still work, albeit maybe just a little bit different.

Automating Renewals

Now that everything’s installed and in place, we’ve got to think about keeping the cert current. Let’s Encrypt certs have a 90-day lifetime, which is substantially shorter than a typical commercial cert. certbot is built to handle automated renewals and can update everything in place without any intervention on your part.

If you try running certbot renew right now, you’ll probably get something like this:

$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/example.com.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal

-------------------------------------------------------------------------------

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/example.com.pro/fullchain.pem (skipped)
No renewals were attempted.
-------------------------------------------------------------------------------

While the cert isn’t due for renewal, we can actually test the renewal process like this:

$ certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/example.com.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
Waiting for verification...
Cleaning up challenges
Dry run: skipping deploy hook command

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/example.com/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/example.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------

This is useful to make sure everything’s in place for automation.

Hooks

You might have noticed the Dry run: skipping deploy hook command line in the output. certbot can run commands or scripts at several stages in its lifecycle. You can either add hooks via flags every time you renew, or you can offload them to executable scripts in /etc/letsencrypt/renewal-hooks.

For this example, all I’d like to do is restart the server process following successful renewals. Assuming we’ve got a local script capable of that called server-reboot, this should add it to certbot’s pipeline.

$ sudo cp ./server-reboot /var/letsencrypt/renewal-hooks/deploy
$ sudo chmod +x /var/letsencrypt/renewal-hooks/deploy

Nginx

/var/letsencrypt/renewal-hooks/deploy/server-reboot
1
2
3
#!/bin/bash

nginx -t && systemctl restart nginx

Apache

/var/letsencrypt/renewal-hooks/deploy/server-reboot
1
2
3
#!/bin/bash

apachectl -t && systemctl restart apachectl

Scripting a Renewal

The official documentation suggests running an automated renewal task at least twice a day (e.g. the CentOS instructions; scroll down). certbot also asks that you run it at a random minute. To make things easier later, let’s isolate our renew command:

/sbin/certbot-renew-everything
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
#!/bin/bash

# Create a temporary file for STDERR
ERROR_LOG=$(mktemp)

# Renew, ignoring STDOUT and piping STDERR to the temp file
/usr/bin/certbot renew > /dev/null 2> "$ERROR_LOG"

if [[ -s "$ERROR_LOG" ]]; then
    mail -s "certbot Renewal Issue" [email protected] < "$ERROR_LOG"
fi

rm -rf "$ERROR_LOG"
$ sudo chmod 'u=rwx,go=' /sbin/certbot-renew-everything

Adding extra flags is straightforward. renew only exits with a nonzero code if the renewal failed (paragraph right above the link), meaning the skipped renewals we saw earlier don’t generate any traffic. They do, however, send many things to STDOUT, which is enough to trigger cron’s mail action. The quiet flag suppresses STDOUT, so you won’t get multiple emails a day letting you know certbot did nothing. If you’re into that you don’t have to use it.

Most of the solutions I’ve seen for the randomness do some cool stuff with advanced PRNG and then pass the result to sleep. There’s nothing wrong with sleep if you’re pausing tasks that don’t actually need to run. Anything that kills the thread kills the task.

at

at provides a much better solution because, via man at,

The at utility shall read commands from standard input and group them together as an at-job, to be executed at a later time.

In other words, at is a single-execution cron. It manages an at queue, most likely accessible via atq, which means random power failure or accidentally nuking a remote session won’t kill the delayed task. Of course that means some setup is required:

$ sudo yum install -y at
$ sudo pkill -f atd
$ sudo systemctl enable atd
$ sudo systemctl start atd
$ sudo systemctl status atd
● atd.service - Job spooling tools
   Loaded: loaded (/usr/lib/systemd/system/atd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-12-17 11:17:15 UTC; 4s ago
 Main PID: 47 (atd)
   CGroup: /system.slice/atd.service
           4747 /usr/sbin/atd -f

Dec 17 11:17:15 examplehost systemd[1]: Started Job spooling tools.
Dec 17 11:17:15 examplehost systemd[1]: Starting Job spooling tools...
  • at is the command itself
  • atd is the at daemon
  • atq is an alias for listing at jobs
  • atrm is an alias for removing at jobs

Block Scheduling

The simplest at solution triggers a script like this NUMBER_OF_DAILY_RUNS times per day.

/sbin/certbot-renew-everything
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash

TASK_FILE=/sbin/certbot-renew-everything

# This assumes you've got some control over the machine's at queues
AT_QUEUE="z"

# The number of times we want the script to run in 24 hours
NUMBER_OF_DAILY_RUNS=2

# The calculated maximum number of minutes per block
MAX_MINUTES=$(( 60 * 24 / $NUMBER_OF_DAILY_RUNS ))

# Create 7 pseudorandom bytes, output as hex
PRN_HEX=$(openssl rand -hex 7)
# The hex is converted to base 10
PRN_TEN=$(( 16#$PRN_HEX ))
# Finally, PRN_TEN is taken mod MAX_MINUTES to fit the domain
PRN_MIN=$(( $PRN_TEN % $MAX_MINUTES ))

# Only execute if this queue is empty
if [[ -z "$( atq -q $AT_QUEUE )" ]]; then
    at "now +${PRN_MIN} min" -q "$AT_QUEUE" -f "$TASK_FILE"
fi

Random Scheduling

A slightly more involved at script calls both the task and itself.

Update 2019/05: Mischa Fiala (@mikfiala) pointed out my original script didn’t work. This updated version should.

/sbin/at-random-renewal
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
#!/bin/bash

# Store original noclobber
ORIGINAL_NOCLOBBER=$( set +o | grep noclobber )
set +o noclobber

# Pull out the PRNG into a function
function openssl_prng {
    MAX=$1
    # Create 7 pseudorandom bytes, output as hex
    PRN_HEX=$(openssl rand -hex 7)
    # The hex is converted to base 10
    PRN_TEN=$(( 16#$PRN_HEX ))
    # Finally, PRN_TEN is taken mod MAX to fit the domain
    PRN_MIN=$(( $PRN_TEN % $MAX ))
    echo $PRN_MIN
}

# Path to renew task
TASK_FILE=/sbin/certbot-renew-everything

# This assumes you've got some control over the machine's at queues
SCRIPT_QUEUE="y"
TASK_QUEUE="z"

# A hard cap on run count to account for unpleasant randomness
ABSOLUTE_RUN_COUNT_MAX=10

# The number of minutes in 24 hours
MINUTES_IN_TWENTY_FOUR_HOURS=$(( 24 * 60 ))

# When to schedule the next renew run
TASK_SLEEP_MINS=$( openssl_prng $MINUTES_IN_TWENTY_FOUR_HOURS )
# Delay scheduling the next self run by an arbitrary amount
SCRIPT_SLEEP_MINS=$(( $TASK_SLEEP_MINS + 30 ))

# Directory to hold active files
RUN_DIR=/var/run/certbot-renew
mkdir -p "$RUN_DIR"
# File to store current date and run count
RUN_COUNT_FILE="$RUN_DIR/count"
touch "$RUN_COUNT_FILE"
# Using awk, load the file
#   * If the dates match, use the loaded run count
#   * If not, reset the count
RUN_COUNT=$( awk '{ if ($1 == strftime("%F")) { print $2; } else { print 0; } }' "$RUN_COUNT_FILE" )

# Get the absolute path to this file
RUN_SCRIPT_PATH_FILE="$RUN_DIR/path"
touch "$RUN_SCRIPT_PATH_FILE"
THIS_SCRIPT=$( [[ -s "$RUN_SCRIPT_PATH_FILE" ]] && cat "$RUN_SCRIPT_PATH_FILE" || readlink -m $0)
rm -rf "$RUN_SCRIPT_PATH_FILE"
if [[ -e "$THIS_SCRIPT" ]]; then
    echo "$THIS_SCRIPT" >| "$RUN_SCRIPT_PATH_FILE"
else
    echo "Unable to find self-reference" | systemd-cat -t certbot-renew-everything
    eval "$ORIGINAL_NOCLOBBER"
    exit 1
fi

# Check that RUN_COUNT is low enough and TASK_QUEUE is empty
if [[ "$RUN_COUNT" -lt "$ABSOLUTE_RUN_COUNT_MAX" ]] && [[ -z "$( atq -q $TASK_QUEUE )" ]]; then
    # Increment RUN_COUNT
    RUN_COUNT=$(( $RUN_COUNT + 1 ))
    # Schedule a renew and run count update
    echo "source $TASK_FILE && (date \"+%F $RUN_COUNT\" >| $RUN_COUNT_FILE)" | at "now +${TASK_SLEEP_MINS} min" -q "$TASK_QUEUE"
fi

# Check that SCRIPT_QUEUE is empty
if [[ -z "$( atq -q $SCRIPT_QUEUE )" ]]; then
    # Schedule a new self run
    at "now +${SCRIPT_SLEEP_MINS} min" -q "$SCRIPT_QUEUE" -f "$THIS_SCRIPT"
fi

# Revert to original noclobber
eval "$ORIGINAL_NOCLOBBER"

Scheduling the Renewal

With or without at, you’ve got to ensure the task is actually being run.

cron

$ sudo crontab -e
crontab -e
1
2
3
4
5
6
7
8
# Add a `MAILTO` that looks like this:
MAILTO=[email protected]
# Add one of the following, depending on how you set it up:
0 0,12 * * * /full/path/to/certbot renew --quiet
# or
0 0,12 * * * /sbin/certbot-renew-everything
# or
0 0,12 * * * /sbin/at-random-renewal

If you’re not changing the time in the script itself, you probably don’t want to use 0 0,12. This launches the task at 00:00 and 12:00 every day. If launching means at assigns a random time, or checks to see if it’s running, those times aren’t a problem. If you’re actually hitting Let’s Encrypt every day at that time, that’s not a great idea.

systemd

(Note: my systemd knowledge is still pretty rudimentary. I’m using to userspace cron. If you see anything I can improve, I’d love to know about it!)

We’re going to define a oneshot unit (example #2):

/etc/systemd/system/certbot-renew.service
1
2
3
4
5
6
7
8
[Unit]
Description=Attempts to renew all certbot certs

[Service]
Type=oneshot
ExecStart=/full/path/to/at/runner
# ExecStart=/sbin/certbot-renew-everything
# ExecStart=/full/path/to/certbot renew --quiet
$ sudo chmod 'ugo=r,u+w' /etc/systemd/system/certbot-renew.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable certbot-renew.service
$ sudo systemctl start certbot-renew.service
$ sudo systemctl status certbot-renew.service

● certbot-renew.service - Attempts to renew all certbot certs
   Loaded: loaded (/etc/systemd/system/certbot-renew.service; static; vendor preset: disabled)
   Active: inactive (dead)

Dec 17 14:50:31 wizardsoftheweb1 systemd[1]: Starting Attempts to renew all certbot certs...
Dec 17 14:50:31 wizardsoftheweb1 systemd[1]: Started Attempts to renew all certbot certs.

To run it regularly, we also create a timer:

/etc/systemd/system/certbot-renew.timer
1
2
3
4
5
6
[Unit]
Description=Run certbot-renew.service every day at 00:00 and 12:00

[Timer]
OnCalendar=*-*-* 00/12:00
Unit=certbot-renew.service
$ sudo chmod 'ugo=r,u+w' /etc/systemd/system/certbot-renew.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable certbot-renew.service
$ sudo systemctl start certbot-renew.service
$ sudo systemctl status certbot-renew.service

● certbot-renew.service - Attempts to renew all certbot certs
   Loaded: loaded (/etc/systemd/system/certbot-renew.service; static; vendor preset: disabled)
   Active: inactive (dead)

Dec 17 14:50:31 wizardsoftheweb1 systemd[1]: Starting Attempts to renew all certbot certs...
Dec 17 14:50:31 wizardsoftheweb1 systemd[1]: Started Attempts to renew all certbot certs.

$ sudo chmod 'ugo=r,u+w' /etc/systemd/system/certbot-renew.timer
$ sudo systemctl daemon-reload
$ sudo systemctl enable certbot-renew.timer
$ sudo systemctl start certbot-renew.timer
$ sudo systemctl status certbot-renew.timer

● certbot-renew.timer - Run certbot-renew.service every day at 00:00 and 12:00.
   Loaded: loaded (/etc/systemd/system/certbot-renew.timer; static; vendor preset: disabled)
   Active: active (waiting) since Sun 2017-12-17 15:03:21 UTC; 4min 3s ago

Dec 17 15:03:21 wizardsoftheweb1 systemd[1]: Started Run certbot-renew.service every day at 00:00 and 12:00.
Dec 17 15:03:21 wizardsoftheweb1 systemd[1]: Starting Run certbot-renew.service every day at 00:00 and 12:00.

$ sudo systemctl list-timers certbot*

NEXT                         LEFT    LAST PASSED UNIT                ACTIVATES
Mon 2017-12-18 00:00:00 UTC  8h left n/a  n/a    certbot-renew.timer certbot-renew.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

Before You Go

Let’s Encrypt is a fantastic service. If you like what they do, i.e. appreciate how accessible they’ve made secure web traffic, please donate. EFF’s certbot is what powers my site (and basically anything I work on these days); consider buying them a beer (it’s really just a donate link but you catch my drift).

I’m still pretty new to the whole CYA legal thing. I really like everything I’ve covered here, and I’ve done my best to respect individual legal policies. If I screwed something up, please send me an email ASAP so I can fix it.

  • The Electronic Frontier Foundation and certbot are covered by EFF’s generous copyright. As far as I know, it’s all under CC BY 3.0 US. I made a few minor tweaks to build the banner image but tried to respect the trademark. I don’t know who the certbot logo artist is but I really wish I did because it’s a fantastic piece of art.
  • Let’s Encrypt is trademarked. Its logo uses CC BY-NC 4.0. I made a few minor tweaks to build the banner image but tried to respect the trademark.
  • I didn’t find anything definitive (other than EULAs) covering Nginx, which doesn’t mean it doesn’t exist. Assets were taken from its press page.
  • Apache content was sourced from its press page. It provides a full trademark policy.