edit: Solved, editing the solution in here at the top for reference if anyone happens to find this via search, original post below.

The caddy config is really simple. Just define a block for cert requests:

(fancy_name) {
  tls {
    client_auth {
      mode require_and_verify
      trusted_ca_cert_file /cert_path/cert_name-CA.crt
      trusted_leaf_cert_file /cert_path/cert_name.crt
    }
  }
}

Import it right before any reverse proxy definition that you want to secure with a client cert:

your_url.com {
  import fancy_name
  reverse_proxy service:port
}

Certificate generation requires some fiddling with openssl. I used these to create a pkcs12 key (no idea whether this is the best way to go about it but it seems reasonable and works) which then can easily be imported into all common browsers:

openssl req -x509 -newkey rsa:4096 -keyout cert_name.key -out cert_name.crt -days 365
openssl req -new -key cert_name.key -out cert_name.csr
openssl x509 -req -days 365 -in cert_name.csr -signkey cert_name.key -out cert_name-CA.crt
cat cert_name.crt cert_name.key > cert_name.pem
openssl pkcs12 -export -out cert_name.p12 -inkey cert_name.key -in cert_name.pem

Done, client certificates working. The files mentioned in the Caddy config need to be placed so that Caddy can access them. The p12 file gets imported client-side (e.g., in the web browser). The other files created by openssl can simply be deleted I think.