Troubleshoot private database access

The VM can reach your database, but an application pod cannot. Logs might show connection refused, a timeout or a failed migration. Check the network path before changing passwords or retrying migrations.

Hakopod pods and the host have different network policies. Public internet egress excludes private addresses, including private RDS, Azure Database and Cloud SQL endpoints. DNS can resolve correctly while TCP remains blocked. An error message alone does not prove a policy problem: routes, firewalls and the database listener can cause similar failures.

Check the connection without credentials#

Run these examples on the self-hosted management server. Replace the database hostname and port. The Python probe opens a TCP connection only; it does not authenticate or change database data.

DB_HOST='db.example.internal'
DB_PORT=5432
python3 - "$DB_HOST" "$DB_PORT" <<'PY'
import socket, sys
host, port = sys.argv[1], int(sys.argv[2])
print('Resolved:', sorted({x[4][0] for x in socket.getaddrinfo(host, port)}))
with socket.create_connection((host, port), timeout=5):
    print('TCP connected')
PY

Use the installer-managed Kubernetes tools to find the affected application namespace and pod:

hpk() {
  sudo /opt/hakopod/tools/k3s kubectl \
    --kubeconfig /etc/hakopod/admin-kubeconfig "$@"
}
hpk get namespaces -l app.kubernetes.io/managed-by=hakopod
APP_NAMESPACE='replace-with-application-namespace'
hpk -n "$APP_NAMESPACE" get pods -L hakopod.io/service
hpk -n "$APP_NAMESPACE" get networkpolicies

For a running Python container, repeat the probe from that pod. Replace the pod name. This uses the container's Python; other images may need their own diagnostic tool. Do not install packages or expose database passwords just to test a socket.

POD='replace-with-running-pod-name'
hpk -n "$APP_NAMESPACE" exec "$POD" -- python -c \
  'import socket,sys; socket.create_connection((sys.argv[1],int(sys.argv[2])),5).close(); print("TCP connected")' \
  "$DB_HOST" "$DB_PORT"

A crashed deployment job cannot be inspected with exec. Ask your installation administrator to use a temporary diagnostic pod with the affected service's policy labels and no application credentials. Remove that diagnostic pod afterward.

Interpret the result#

For PostgreSQL TLS, keep sslmode=verify-full and the provider's CA file in sslrootcert. The hostname must match the certificate. Do not disable certificate verification to work around a networking failure.

Configure native private egress#

The installer release v0.1.0-alpha.39 supports private_egress on self-hosted installations. The installation administrator approves a named destination; each allowed service then references it in application TOML. Managed Cloud does not accept these grants. Database secrets do not grant network access.

Create /etc/hakopod/private-egress.toml on the management server. Replace the example scope, service names and subnets with the actual database destination. Approve only the required database subnets and TCP port:

schema_version = 1

[[bindings]]
name = "orders-db"
project = "shop"
environment = "production"
application = "orders"
services = ["api", "setup", "worker"]
cidrs = ["10.20.30.0/24", "10.20.31.0/24"]
ports = [5432]

For the standard installer, protect the file and configure the API service to load it. If the override file below already exists, merge the environment setting into it instead of overwriting it:

sudo chown root:hakopod-api /etc/hakopod/private-egress.toml
sudo chmod 0640 /etc/hakopod/private-egress.toml
sudo mkdir -p /etc/systemd/system/hakopod-api.service.d
sudo tee /etc/systemd/system/hakopod-api.service.d/private-egress.conf >/dev/null <<'UNIT'
[Service]
Environment="HAKOPOD_PRIVATE_EGRESS_FILE=/etc/hakopod/private-egress.toml"
UNIT
sudo systemctl daemon-reload
sudo systemctl restart hakopod-api

Custom installations should set the same environment variable and give the API process read access to the protected file.

Add the reference to each service that needs access, keeping its other settings:

[services.api]
# Keep the existing image, port and other service settings.
private_egress = ["orders-db"]

Repeat the reference in the existing setup and worker tables if they need the database. The grant must match the project, environment, application and service exactly. It works for services, deployment jobs and scheduled jobs, including services on internal networks. Review and deploy the updated application, then repeat the pod probe before retrying a failed migration.

Bindings support private RFC1918 IPv4 CIDRs of /16 or narrower. Cluster pod and Service ranges are rejected; do not approve node, control-plane or unrelated workload addresses. Hakopod does not derive these ranges from DNS. Routes, database firewalls, credentials and TLS remain separate requirements.

Change or remove native access#

To remove access, remove the service reference and successfully deploy the application, then remove the unused administrator binding. Changing the file requires an API restart and a deployment to update existing network policies. Restarting the API alone does not revoke access from running pods. For emergency revocation, an administrator must also update the existing network policy or external firewall.

For failover, approve the reviewed database subnets or update the binding when the destination changes. A /32 rule permits only one IPv4 address. If an older operator policy also grants access, remove that specific policy only after the native rule is deployed and verified; otherwise it can keep granting access after the native reference is removed.

See the release source guide for binding limits and deployment behavior. Its alpha.13 candidate note predates this installer release.

Operator rule for alpha.12 and earlier#

v0.1.0-alpha.12 and earlier do not have application-level private database egress configuration. The installation administrator can add a separate Kubernetes NetworkPolicy for this case. Do not delete the default deny policy or allow the whole VPC.

The example below permits only the API, setup job and worker in the selected application namespace to reach 10.20.30.15:5432. Replace the example IP, port and service names with reviewed values. Keep the database private. Security groups, routes and database authorization still apply.

Inspect existing policies and save this as a new, uniquely named file such as operator-private-database.yaml. Review it before applying:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: operator-private-database
  labels:
    app.kubernetes.io/managed-by: hakopod-operator
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/managed-by: hakopod
    matchExpressions:
      - key: hakopod.io/service
        operator: In
        values: [api, setup, worker]
  policyTypes: [Egress]
  egress:
    - to:
        - ipBlock:
            cidr: 10.20.30.15/32
      ports:
        - protocol: TCP
          port: 5432

hpk -n "$APP_NAMESPACE" diff -f operator-private-database.yaml
# kubectl diff exits 1 when the reviewed manifest differs.
hpk -n "$APP_NAMESPACE" apply -f operator-private-database.yaml

Choose a new policy name; do not overwrite another operator's policy. The rule adds access only in this namespace. The operator owns and maintains it; Hakopod's reconciler does not. Keep the reviewed manifest with your installation backups.

Repeat the pod probe. Before retrying the deployment, confirm connectivity and database settings, and check whether the failed attempt left any migration changes. This test does not itself rerun migrations.

Maintain or remove the legacy policy#

A /32 rule allows one IPv4 address. A database failover can change that address. An administrator may instead approve the actual database subnet CIDRs and the database port, provided those subnets contain only intended destinations. Do not include cluster pod or Service networks (10.42.0.0/16 and 10.43.0.0/16), node/control-plane addresses, metadata endpoints or unrelated workloads.

Review the rule after database/network changes. Kubernetes NetworkPolicy does not track DNS names. Remove only the workaround policy you created when it is no longer needed:

hpk -n "$APP_NAMESPACE" delete networkpolicy operator-private-database

Deleting it can immediately interrupt database access. Confirm that the application no longer needs it, or that a verified replacement grant is active first.