Troubleshoot private database access

The VM can reach your database, but an application pod cannot. Logs might show connection refused, a timeout or a failed migration. Check the network path before changing passwords or retrying migrations.

Hakopod pods and the host have different network policies. Public internet egress excludes private addresses, including private RDS, Azure Database and Cloud SQL endpoints. DNS can resolve correctly while TCP remains blocked. An error message alone does not prove a policy problem: routes, firewalls and the database listener can cause similar failures.

Check the connection without credentials#

Run these examples on the self-hosted management server. Replace the database hostname and port. The Python probe opens a TCP connection only; it does not authenticate or change database data.

DB_HOST='db.example.internal'
DB_PORT=5432
python3 - "$DB_HOST" "$DB_PORT" <<'PY'
import socket, sys
host, port = sys.argv[1], int(sys.argv[2])
print('Resolved:', sorted({x[4][0] for x in socket.getaddrinfo(host, port)}))
with socket.create_connection((host, port), timeout=5):
    print('TCP connected')
PY

Use the installer-managed Kubernetes tools to find the affected application namespace and pod:

hpk() {
  sudo /opt/hakopod/tools/k3s kubectl \
    --kubeconfig /etc/hakopod/admin-kubeconfig "$@"
}
hpk get namespaces -l app.kubernetes.io/managed-by=hakopod
APP_NAMESPACE='replace-with-application-namespace'
hpk -n "$APP_NAMESPACE" get pods -L hakopod.io/service
hpk -n "$APP_NAMESPACE" get networkpolicies

For a running Python container, repeat the probe from that pod. Replace the pod name. This uses the container's Python; other images may need their own diagnostic tool. Do not install packages or expose database passwords just to test a socket.

POD='replace-with-running-pod-name'
hpk -n "$APP_NAMESPACE" exec "$POD" -- python -c \
  'import socket,sys; socket.create_connection((sys.argv[1],int(sys.argv[2])),5).close(); print("TCP connected")' \
  "$DB_HOST" "$DB_PORT"

A crashed deployment job cannot be inspected with exec. Ask your installation administrator to use a temporary diagnostic pod with the affected service's policy labels and no application credentials. Remove that diagnostic pod afterward.

Interpret the result#

For PostgreSQL TLS, keep sslmode=verify-full and the provider's CA file in sslrootcert. The hostname must match the certificate. Do not disable certificate verification to work around a networking failure.

Current releases: a narrow operator rule#

v0.1.0-alpha.12 and earlier do not have application-level private database egress configuration. The installation administrator can add a separate Kubernetes NetworkPolicy for this case. Do not delete the default deny policy or allow the whole VPC.

The example below permits only the API, setup job and worker in the selected application namespace to reach 10.20.30.15:5432. Replace the example IP, port and service names with reviewed values. Keep the database private. Security groups, routes and database authorization still apply.

Inspect existing policies and save this as a new, uniquely named file such as operator-private-database.yaml. Review it before applying:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: operator-private-database
  labels:
    app.kubernetes.io/managed-by: hakopod-operator
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/managed-by: hakopod
    matchExpressions:
      - key: hakopod.io/service
        operator: In
        values: [api, setup, worker]
  policyTypes: [Egress]
  egress:
    - to:
        - ipBlock:
            cidr: 10.20.30.15/32
      ports:
        - protocol: TCP
          port: 5432

hpk -n "$APP_NAMESPACE" diff -f operator-private-database.yaml
# kubectl diff exits 1 when the reviewed manifest differs.
hpk -n "$APP_NAMESPACE" apply -f operator-private-database.yaml

Choose a new policy name; do not overwrite another operator's policy. The rule is additive, applies only in this namespace, and is deliberately owned by the operator rather than Hakopod's reconciler. Keep the reviewed manifest with your installation backups.

Repeat the pod probe. Only after connectivity and database configuration are correct should you retry the failed deployment, taking account of any migration side effects. This test does not itself rerun migrations.

Failover and removing access#

A /32 rule allows one IPv4 address. A database failover can change that address. An administrator may instead approve the actual database subnet CIDRs and the database port, provided those subnets contain only intended destinations. Do not include cluster pod or Service networks (10.42.0.0/16 and 10.43.0.0/16), node/control-plane addresses, metadata endpoints or unrelated workloads.

Review the rule after database/network changes. Kubernetes NetworkPolicy does not track DNS names. Remove only the workaround policy you created when it is no longer needed:

hpk -n "$APP_NAMESPACE" delete networkpolicy operator-private-database

Deleting it can immediately interrupt database access. Confirm that the application no longer needs it, or that a verified replacement grant is active first.

Native configuration is being added#

The next implementation adds named administrator-approved destinations that a service can reference with private_egress = ["orders-db"]. Hakopod will validate the exact service scope and reconcile its network rule with the application revision. This is unreleased; do not paste that field into alpha.12 or older installations.

The source guide describes the operator file, limits and rollout requirements once merged. No network access will be inferred from database secrets. Managed Cloud support is outside this initial self-hosted feature.