Visualization service - Setup Superset on K8s

Deploying Apache Superset on Kubernetes with PostgreSQL Support

This tutorial covers the complete process of deploying Apache Superset on a Kubernetes cluster with PostgreSQL database driver support. We’ll walk through the challenges we faced and how we solved them.

Prerequisites

• Kubernetes cluster (we used DigitalOcean Kubernetes)
• kubectl configured to access your cluster
• Domain name with DNS access
• NGINX Ingress Controller installed
• cert-manager for SSL certificates

Overview

We’ll deploy Superset using custom Kubernetes manifests instead of the official Helm chart, which gives us more control over the configuration and allows us to add PostgreSQL support.

Step 1: Create the Superset Namespace and Secret

First, let’s create a dedicated namespace and secret for Superset:

1# superset-deployment.yaml2apiVersion: v13kind: Namespace4metadata:5  name: superset6---7apiVersion: v18kind: Secret9metadata:10  name: superset-secret11  namespace: superset12type: Opaque13data:14  SUPERSET_SECRET_KEY: # your-secret-key-change-this-in-production

Step 2: Create the Superset Deployment

The deployment is the most complex part. Here’s our configuration:

1apiVersion: apps/v12kind: Deployment3metadata:4  name: superset5  namespace: superset6spec:7  replicas: 18  selector:9    matchLabels:10      app: superset11  template:12    metadata:13      labels:14        app: superset15    spec:16      containers:17      - name: superset18        image: apache/superset:latest19        ports:20        - containerPort: 808821        env:22        - name: SUPERSET_SECRET_KEY23          valueFrom:24            secretKeyRef:25              name: superset-secret26              key: SUPERSET_SECRET_KEY27        - name: X_FRAME_OPTIONS28          value: "ALLOWALL"29        command: ["/bin/bash"]30        args:31          - -c32          - |33            # Install PostgreSQL driver in user directory34            pip install --user psycopg2-binary35            # Add user site-packages to Python path36            export PYTHONPATH="/app/superset_home/.local/lib/python3.10/site-packages:$PYTHONPATH"37            # Initialize Superset38            superset db upgrade39            superset fab create-admin \40              --username admin \41              --firstname Admin \42              --lastname User \43              --email [email protected] \44              --password your-password-change-this-in-production45            superset init46            # Start Superset47            superset run -h 0.0.0.0 -p 8088 --with-threads --reload --debugger48        resources:49          requests:50            memory: "256Mi"51            cpu: "100m"52          limits:53            memory: "512Mi"54            cpu: "200m"

Step 3: Create the Service

1apiVersion: v12kind: Service3metadata:4  name: superset-service5  namespace: superset6spec:7  selector:8    app: superset9  ports:10  - port: 808811    targetPort: 808812  type: ClusterIP

Step 4: Create the Ingress

1apiVersion: networking.k8s.io/v12kind: Ingress3metadata:4  name: superset-ingress5  namespace: superset6  annotations:7    cert-manager.io/cluster-issuer: "letsencrypt-prod"8    nginx.ingress.kubernetes.io/ssl-redirect: "true"9spec:10  ingressClassName: nginx11  tls:12  - hosts:13    - superset.do.zeelu.me14    secretName: superset-tls15  rules:16  - host: superset.do.zeelu.me17    http:18      paths:19      - path: /20        pathType: Prefix21        backend:22          service:23            name: superset-service24            port:25              number: 8088

Step 5: Apply the Configuration

1kubectl apply -f superset-deployment.yaml

Challenges and Solutions

Challenge 1: PostgreSQL Driver Missing

Problem: When trying to connect to PostgreSQL databases, Superset showed the error:

1ERROR: Could not load database driver: PostgresEngineSpec

Root Cause: The default Superset Docker image doesn’t include the PostgreSQL driver (psycopg2).

Solution: We modified the deployment to install the PostgreSQL driver during container startup:

1# Install PostgreSQL driver in user directory2pip install --user psycopg2-binary3# Add user site-packages to Python path4export PYTHONPATH="/app/superset_home/.local/lib/python3.10/site-packages:$PYTHONPATH"

Challenge 2: Permission Issues

Problem: When trying to install the PostgreSQL driver, we encountered permission errors:

1PermissionError: [Errno 13] Permission denied: '/app/.venv/lib/python3.10/site-packages/psycopg2_binary.libs'

Root Cause: The Superset container runs as a non-root user (superset) for security, but the virtual environment directory is owned by root.

Solution: We installed the driver in the user directory using pip install --user and added the user site-packages to the Python path.

Challenge 3: Resource Constraints

Problem: The initial deployment failed with insufficient resources:

1Insufficient cpu, 1 Insufficient memory

Root Cause: The single-node DigitalOcean cluster had limited resources.

Solution: We reduced the resource requests and limits:

1resources:2  requests:3    memory: "256Mi"4    cpu: "100m"5  limits:6    memory: "512Mi"7    cpu: "200m"

Challenge 4: SSL Certificate Issues

Problem: SSL certificates were taking a long time to be issued.

Root Cause: DNS propagation delays - the cluster’s DNS resolver couldn’t resolve the new domain immediately.

Solution: We waited for DNS propagation and used HTTP access temporarily while the certificate was being issued.

Challenge 5: Iframe Embedding Blocked

Problem: When trying to embed Superset in an iframe, browsers showed the error:

1Refused to display 'https://superset.do.zeelu.me/' in a frame because it set 'X-Frame-Options' to 'sameorigin'.

Root Cause: Superset by default sets X-Frame-Options: sameorigin which prevents embedding in iframes from different domains.

Solution: We added the environment variable to allow iframe embedding:

1env:2- name: X_FRAME_OPTIONS3  value: "ALLOWALL"

DNS Configuration

Add an A record in your DNS provider:

• Name: superset (for superset.do.zeelu.me)
• Value: Your load balancer IP (e.g., 104.248.105.26)
• TTL: Default

Accessing Superset

Once deployed, you can access Superset at your domain/ip.

Or you can try out mine at:

• URL: https://superset.do.zeelu.me
• Username: guestuser
• Password: guestuser

Verifying PostgreSQL Support

To verify that PostgreSQL support is working:

1. Log into Superset
2. Go to Settings → Database Connections
3. Click + Database
4. Select PostgreSQL from the database type dropdown
5. You should now be able to configure PostgreSQL connections without the driver error

Production Considerations

For production deployments, consider:

1. Security: Change default passwords and use proper secret management
2. Persistence: Use a proper database (PostgreSQL/MySQL) instead of SQLite
3. Scaling: Configure multiple replicas and proper resource limits
4. Monitoring: Add health checks and monitoring
5. Backup: Implement regular database backups

Troubleshooting

Check Pod Status

1kubectl get pods --namespace superset

View Logs

1kubectl logs <pod-name> --namespace superset

Check SSL Certificate

1kubectl get certificates --namespace superset

Conclusion

Deploying Superset on Kubernetes with PostgreSQL support requires careful attention to container permissions, resource constraints, and DNS configuration. By using custom manifests instead of Helm charts, we gained the flexibility to install additional packages and configure the deployment exactly as needed.

The key was understanding that modern containers run as non-root users and installing packages in the user directory while ensuring the Python path includes the user site-packages directory.