[merlimat]

Motivation

When the chart deploys Oxia as the metadata store, Oxia's cluster-internal endpoints are more exposed than they need to be:

• The <release>-pulsar-oxia ClusterIP Service publishes the servers' internal endpoint (6649) and their metrics endpoint, not just the public endpoint. The internal endpoint serves the RPCs the Oxia servers use for replication and the coordinator uses to manage the shards, and it has no authentication by default. Nothing in the chart uses the extra ports through this Service: the Pulsar components connect to the public endpoint through the headless <release>-pulsar-oxia-svc Service, and the PodMonitors scrape the pods directly.
• There is no NetworkPolicy, so any pod in the cluster can reach the internal endpoints of the servers and the coordinator, and the coordinator's admin endpoint (6651), through pod IPs or the headless Service.
• tls.oxia.enabled has been in values.yaml since Oxia support was added (Add support for using Oxia as the metadata store for Pulsar and BookKeeper #544), but no template uses it.

For example, with Oxia enabled, any pod in any namespace can open a connection to <release>-pulsar-oxia:6649 or <release>-pulsar-oxia-server-0.<release>-pulsar-oxia-svc:6649 and call the internal RPCs.

Modifications

• oxia-server-service-public.yaml: the <release>-pulsar-oxia Service only exposes the public port. The headless Service is unchanged.

• New oxia-networkpolicy.yaml, controlled by oxia.networkPolicy.enabled (default true). It selects the Oxia server and coordinator pods and allows:

  • the public endpoint from anywhere (oxia.networkPolicy.public.from can restrict it),
  • the server and coordinator metrics endpoints from anywhere (oxia.networkPolicy.metrics.from),
  • the server internal, coordinator internal and coordinator admin (6651) endpoints only from the Oxia pods of the release, plus oxia.networkPolicy.internal.from.

  The Pulsar components wait for Oxia with nslookup and connect to the public endpoint, and the probes are exec probes, so the policy doesn't affect them. It needs a CNI plugin that enforces NetworkPolicy.

• tls.oxia now takes effect, together with tls.enabled:

  • tls-certs-internal.yaml creates the <release>-tls-oxia Certificate through pulsar.cert.template, for the server pods and the headless Service. createCert: false, dnsNames, ipAddresses, secretAnnotations and every tls.common.sanMode work as for the other components.
  • The servers and the coordinator mount it at /pulsar/certs/oxia, and the CA at /pulsar/certs/ca.
  • The servers get --internal-tls-* for their internal endpoint and --peer-tls-* for replication. The coordinator gets --peer-tls-* for its calls to the servers.
  • tls.oxia.clientAuth adds --internal-tls-client-auth (mTLS).
  • The servers share one certificate, so the client side verifies it against the headless Service name (--peer-tls-server-name). The certificate contains that name in every SAN mode.
  • The public endpoint stays plaintext, because the Oxia Java client in Pulsar only trusts the JVM's default CA certificates, so it can't verify a certificate from the chart's CA.
  • oxia health only speaks plaintext, so with TLS on, the server probes use the public endpoint, which serves the same health service.

• README: a new "Oxia network access and TLS" section.

• CI:

  • a new "Oxia TLS" scenario (.ci/clusters/values-oxia-tls.yaml, with mTLS),
  • Oxia added to .ci/test-certificate-san-modes.sh for all three SAN modes,
  • the NetworkPolicy from lists and Oxia mTLS added to .ci/templates-all-values.yaml.

Upgrade notes:

• Anything outside the release that connects to the Oxia internal or admin endpoints, or that uses the internal or metrics ports through <release>-pulsar-oxia, needs to be added to oxia.networkPolicy.internal.from or switched to the headless Service. NetworkPolicies are additive, so a release whose Oxia pods are already selected by stricter policies gets this policy's allowed sources added. Set oxia.networkPolicy.enabled: false to keep the previous behavior.
• Turning tls.oxia on or off in an existing release restarts the Oxia pods. Servers on the new setting can't replicate with servers on the old one until all of them have restarted.

Verifying this change

• Make sure that the change passes the CI checks.

• helm lint.

• helm template diffed against master:

  • Defaults (ZooKeeper) and the all-values render are unchanged.
  • With Oxia, only the trimmed Service and the new NetworkPolicy differ.
  • With tls.oxia, the Oxia StatefulSet and Deployment add only the TLS flags, the cert mounts and the probe port, and there is a new Certificate.

• .ci/test-certificate-san-modes.sh passes. A negative control with a wrong Oxia service name fails.

• I haven't run this chart on a cluster yet. The new "Oxia TLS" scenario covers a fresh install with mTLS, and the existing Oxia scenario now runs with the NetworkPolicy. The same NetworkPolicy and internal-endpoint mTLS setup was tested on kind with Oxia 0.16.11 in the equivalent oxia-db/helm-charts change (feat: restrict network exposure and add first-class TLS oxia-db/helm-charts#15): the cluster forms, and from other namespaces 6648 and metrics stay reachable while 6649 and 6651 are blocked. Plaintext connections, and TLS connections without a client certificate, are rejected on 6649.

[lhotari]

Thanks for tightening up the Oxia exposure and wiring up tls.oxia; the change is well scoped and the new CI scenario is a nice touch.

I rendered the chart at this head and checked the policy and the TLS wiring against the pod labels, the other templates and Oxia 0.16.11:

• The default NetworkPolicy does not block anything the chart itself needs. The component labels in the selector match the server and coordinator pods, the Pulsar components and init jobs reach Oxia only on the public port through the headless Service, the probes are exec probes, and omitting from on the public and metrics rules means "allow from anywhere". Server-to-server, coordinator-to-server and the 6651 admin port are limited to the release's Oxia pods.
• --internal-tls-*, --internal-tls-client-auth, --peer-tls-* and --peer-tls-server-name all exist in Oxia 0.16.11. The certificate always contains the headless Service name, so the server-name override verifies in every SAN mode (.ci/test-certificate-san-modes.sh passes locally), and Oxia reloads the key pair on each handshake, so cert-manager renewals are picked up without restarts.
• oxia health is plaintext only, and the health and readiness services are registered on the public server as well, so moving the probes to the public port when TLS is on works.
• The 4.8.0 upgrade section of the README is left untouched.

There is one usability point inline on oxia.networkPolicy.public.from, and two smaller things:

1. The default-on policy and the trimmed <release>-pulsar-oxia Service are behavior changes for existing Oxia releases. Both are explained in the PR description but not in the README upgrade guidance. Since the 4.8.0 section is frozen, I assume a short note belongs in the section for the next chart version once it exists. It could also mention that a release that already has tls.oxia.enabled: true (a no-op until now) starts using TLS, and restarts the Oxia pods, on upgrade.
2. With tls.oxia on, the coordinator's internal endpoint and the 6651 admin endpoint remain plaintext and are only protected by the NetworkPolicy. I assume that is intentional because nothing in the chart connects to them over TLS, but a sentence in the README section saying so would avoid the wrong impression that all Oxia process-to-process traffic is encrypted.

The rolling restart when toggling tls.oxia on an existing release (servers on the new setting cannot replicate with servers on the old one) is described but, as you say, not tested on a cluster. A short manual check of that path before release would be reassuring.