nixos/acme: adjust renewal timer options

The current weekly setting causes every NixOS server to try to renew
its certificate at midnight on the dot on Monday. This contributes to
the general problem of periodic load spikes for Let's Encrypt; NixOS
is probably not a major contributor to that problem, but we can lead by
example by picking good defaults here.

The values here were chosen after consulting with @yuriks, an SRE at
Let's Encrypt:

* Randomize the time certificates are renewed within a 24 hour period.

* Check for renewal every 24 hours, to ensure the certificate is always
  renewed before an expiry notice is sent out.

* Increase the AccuracySec (thus lowering the accuracy(!)), so that
  systemd can coalesce the renewal with other timers being run.

  (You might be worried that this would defeat the purpose of the time
  skewing, but systemd is documented as avoiding this by picking a
  random time.)

1 files changed, 15 insertions, 4 deletions

diff --git a/nixos/modules/security/acme.nix b/nixos/modules/security/acme.nix
index 7da6666f79c6..dfd0c2cc2e3c 100644
--- a/nixos/modules/security/acme.nix
+++ b/nixos/modules/security/acme.nix
@@ -174,7 +174,7 @@ in
 
       renewInterval = mkOption {
         type = types.str;
-        default = "weekly";
+        default = "daily";
         description = ''
           Systemd calendar expression when to check for renewal. See
           <citerefentry><refentrytitle>systemd.time</refentrytitle>
@@ -399,7 +399,17 @@ in
       systemd.tmpfiles.rules =
         map (data: "d ${data.webroot}/.well-known/acme-challenge - ${data.user} ${data.group}") (filter (data: data.webroot != null) (attrValues cfg.certs));
 
-      systemd.timers = flip mapAttrs' cfg.certs (cert: data: nameValuePair
+      systemd.timers = let
+        # Allow systemd to pick a convenient time within the day
+        # to run the check.
+        # This allows the coalescing of multiple timer jobs.
+        # We divide by the number of certificates so that if you
+        # have many certificates, the renewals are distributed over
+        # the course of the day to avoid rate limits.
+        numCerts = length (attrNames cfg.certs);
+        _24hSecs = 60 * 60 * 24;
+        AccuracySec = "${toString (_24hSecs / numCerts)}s";
+      in flip mapAttrs' cfg.certs (cert: data: nameValuePair
         ("acme-${cert}")
         ({
           description = "Renew ACME Certificate for ${cert}";
@@ -408,8 +418,9 @@ in
             OnCalendar = cfg.renewInterval;
             Unit = "acme-${cert}.service";
             Persistent = "yes";
-            AccuracySec = "5m";
-            RandomizedDelaySec = "1h";
+            inherit AccuracySec;
+            # Skew randomly within the day, per https://letsencrypt.org/docs/integration-guide/.
+            RandomizedDelaySec = "24h";
           };
         })
       );