CLI swarm CA rotation with external (CFSSL) CA seems to be (very) broken

[sa-lr]

With Docker 18.03, the following:

docker swarm ca --rotate --ca-cert /path/to/ca.pem --external-ca protocol=cfssl,url=https://cfsslserver.example/api/v1/cfssl/sign

fails with:

Error response from daemon: rpc error: code = InvalidArgument desc = there must be at least one valid, reachable external CA corresponding to the desired CA certificate

Running dockerd in debug mode shows that the underlying API call to https://docs.docker.com/engine/api/v1.37/#operation/SwarmUpdate has an empty ExternalCAs.CACert, and instead the --ca-cert param. populates SigningCACert.

Calling the API directly with the CFSSL CA in the CACert field results in successful cert. rotation.

Is there an undocumented param. that maps to CACert, or is the CLI just completely broken for this purpose?

Some other issues:

• Omitting the --ca-cert param. entirely results in the external CA param. being completely ignored, and the CA rotating to another internal CA (also bad/unexpected behaviour)
• It's not clear from the documentation at https://docs.docker.com/engine/reference/commandline/swarm_ca/ that the endpoint should be the full path to CFSSL's /sign
• CFSSL is not mentioned anywhere on the above doc. page, nor is the KV format of the --external-ca param.
• Specifying a cert. + key (as an externally generated intermediate CA pair signed by CFSSL) "works" in that the CA seems to rotate and is listed in docker info as having CFSSL as an external CA (and the swarm functions), but then on next restart of the swarm nodes, the nodes can't communicate with each other to re-establish the swarm, throwing a generic "bad TLS" error.

[thaJeztah]

/cc @cyli

[cyli]

@sa-lr Thank you for reporting and debugging these issues.

Running dockerd in debug mode shows that the underlying API call to https://docs.docker.com/engine/api/v1.37/#operation/SwarmUpdate has an empty ExternalCAs.CACert, and instead the --ca-cert param. populates SigningCACert

You are correct the CLI is not setting the ExternalCA on rotate, and it should be - I'll submit a PR to fix that.

Omitting the --ca-cert param. entirely results in the external CA param. being completely ignored, and the CA rotating to another internal CA (also bad/unexpected behaviour)

Thanks, that is also problematic. I'll make sure that's addressed.

Specifying a cert. + key (as an externally generated intermediate CA pair signed by CFSSL) "works" in that the CA seems to rotate and is listed in docker info as having CFSSL as an external CA (and the swarm functions), but then on next restart of the swarm nodes, the nodes can't communicate with each other to re-establish the swarm, throwing a generic "bad TLS" error.

Would you happen have any daemon logs with more complete error messages on startup?

Another issue is that doc does not make clear that the external CA URL must be HTTPS and must have a TLS certificate that is signed by the CA certificate provided by --ca-cert.