DEV Community

Tim Kang
Tim Kang

Posted on

Migrating from Terraform/Helm to Database-Driven Kubernetes Without Deleting Anything

#kubernetes #devops #lynq

So you've seen how database-driven automation works, maybe even tried the Killercoda demo, and now you're thinking: "cool, but I already have a hundred tenants running. I can't just blow everything away and start over."

Yeah. That's a real problem.

This post is about how to migrate existing resources managed by Terraform, Helm, Kustomize, or whatever else you're using, over to Lynq without deleting anything. The goal is zero downtime, no data loss, and a safe rollback path if things go sideways.

Before we start

I'm assuming you've already:

  • Set up your MySQL database with the node table
  • Created a LynqHub pointing to that database
  • Written a basic LynqForm and tested it with a fresh node
  • Verified that new nodes provision correctly

If you haven't done that yet, check out the quickstart first. This guide is specifically about taking over existing resources, not creating new ones.

The strategy

Here's the high-level approach:

  1. Configure your LynqForm to generate the exact same resource names as your existing ones
  2. Use conservative policies as safety nets
  3. Test with one node first to verify conflict detection works
  4. Remove ownership from your old tool (Terraform state, Helm release, etc.)
  5. Let Lynq take over ownership
  6. Repeat for remaining nodes
  7. Gradually relax safety policies once stable

Let's walk through each step.

Step 1: match your existing resource names

This is crucial. Your LynqForm templates need to produce the exact same resource names that already exist in the cluster.

Say your existing deployment is named acme-corp-app in namespace acme-corp. Your template needs to render to exactly that:

deployments:
  - id: app
    nameTemplate: "{{ .uid }}-app"
    namespaceTemplate: "{{ .uid }}"
    spec:
      apiVersion: apps/v1
      kind: Deployment
      # ...
Enter fullscreen mode Exit fullscreen mode

If your database has uid = "acme-corp", this renders to acme-corp-app in namespace acme-corp. Perfect match.

Double check your naming conventions. If Terraform was using underscores and Lynq templates use dashes, you'll create duplicate resources instead of taking over the existing ones.

Step 2: configure safety-first policies

For migration, start with the most conservative settings:

deployments:
  - id: app
    nameTemplate: "{{ .uid }}-app"
    conflictPolicy: Stuck      # don't force takeover yet
    deletionPolicy: Retain     # never auto-delete, even if something goes wrong
    creationPolicy: WhenNeeded
    spec:
      # ...
Enter fullscreen mode Exit fullscreen mode

Why these settings?

conflictPolicy: Stuck is your early warning system. When Lynq tries to apply a resource that's already owned by something else (like Terraform or Helm), it will stop and emit an event instead of forcing through. This lets you verify that Lynq is actually targeting the right resources.

deletionPolicy: Retain is your safety net. Even if you accidentally delete a LynqNode or mess up the hub config, the actual kubernetes resources stay in the cluster. You can always recover.

Apply this to every resource in your template. Yes, all of them.

Step 3: test with a single node first

Don't migrate everything at once. Pick one tenant/node and try it first.

Insert or activate the row in your database:

UPDATE nodes SET is_active = true WHERE node_id = 'acme-corp';
Enter fullscreen mode Exit fullscreen mode

Now watch what happens:

kubectl get lynqnodes -w
Enter fullscreen mode Exit fullscreen mode

You should see a LynqNode created. Check its events:

kubectl describe lynqnode acme-corp-web-app
Enter fullscreen mode Exit fullscreen mode

If your existing resources match the template output, you'll see ResourceConflict events. This is actually what we want at this stage. It confirms Lynq is finding and targeting the right resources.

The event message tells you exactly what's conflicting:

Resource conflict detected for acme-corp/acme-corp-app (Kind: Deployment, Policy: Stuck). 
Another controller or user may be managing this resource. Consider using ConflictPolicy=Force 
to take ownership or resolve the conflict manually. 
Error: Apply failed with 1 conflict: conflict with "helm" using apps/v1: .spec.replicas
Enter fullscreen mode Exit fullscreen mode

This tells you:

  • Which resource: acme-corp/acme-corp-app
  • Current owner: helm
  • Conflicting field: .spec.replicas

Step 4: review and fix template mismatches

Sometimes the conflict message reveals that your LynqForm doesn't quite match the existing resource. Maybe your template sets replicas: 2 but the existing deployment has replicas: 5 because of HPA.

You have a few options:

Option A: Update your template to match

If the difference is intentional (like HPA managing replicas), don't set that field in your template, or use ignoreFields to skip it during reconciliation.

Option B: Accept the difference

If you want Lynq to enforce a new value, that's fine. Just be aware the resource will change when you force takeover.

Option C: Update the database

If the value should come from your database, add it to extraValueMappings and use it in the template.

The key is understanding what will change before you flip the switch.

Step 5: remove ownership from your old tool

Now comes the actual migration. You need to tell your old tool to stop managing these resources without deleting them.

For Terraform:

# Remove from state without destroying
terraform state rm kubernetes_deployment.acme_corp_app
terraform state rm kubernetes_service.acme_corp_svc
# repeat for all resources
Enter fullscreen mode Exit fullscreen mode

For Helm:

# Uninstall release but keep resources
helm uninstall acme-corp-release --keep-history

# Or if you want to be extra safe, just delete the release secret
kubectl delete secret -l owner=helm,name=acme-corp-release
Enter fullscreen mode Exit fullscreen mode

For Kustomize/kubectl:

If you were just applying manifests directly, there's no state to remove. The resources exist, they're just not tracked by anything. Lynq can take over directly.

For ArgoCD/Flux:

Remove the Application or Kustomization CR, or exclude those resources from sync. The actual resources stay in cluster.

After this step, the resources exist in kubernetes but nothing is actively managing them. They're orphaned, which is exactly what we want temporarily.

Step 6: let lynq take ownership

Now update your LynqForm to force takeover:

deployments:
  - id: app
    nameTemplate: "{{ .uid }}-app"
    conflictPolicy: Force      # changed from Stuck
    deletionPolicy: Retain     # keep this for now
    spec:
      # ...
Enter fullscreen mode Exit fullscreen mode

Apply the updated LynqForm:

kubectl apply -f lynqform.yaml
Enter fullscreen mode Exit fullscreen mode

The next reconciliation will use Server-Side Apply with force=true to take ownership. Check the LynqNode status:

kubectl get lynqnode acme-corp-web-app -o yaml
Enter fullscreen mode Exit fullscreen mode

You should see:

status:
  desiredResources: 3
  readyResources: 3
  failedResources: 0
  appliedResources:
    - "Deployment/acme-corp/acme-corp-app@app"
    - "Service/acme-corp/acme-corp-svc@svc"
Enter fullscreen mode Exit fullscreen mode

No more conflicts. Lynq now owns these resources.

Verify by checking the resource's managedFields:

kubectl get deployment acme-corp-app -n acme-corp -o yaml | grep -A5 managedFields
Enter fullscreen mode Exit fullscreen mode

You should see manager: lynq in there.

Step 7: repeat for remaining nodes

Once you've confirmed the first node works, migrate the rest. You can do this gradually:

-- Migrate in batches
UPDATE nodes SET is_active = true WHERE region = 'us-east-1';
-- Wait, verify
UPDATE nodes SET is_active = true WHERE region = 'us-west-2';
-- And so on
Enter fullscreen mode Exit fullscreen mode

Monitor the LynqHub status to track progress:

kubectl get lynqhub my-hub -o yaml
Enter fullscreen mode Exit fullscreen mode 
status:
  referencingTemplates: 1
  desired: 150
  ready: 148
  failed: 2
Enter fullscreen mode Exit fullscreen mode

Investigate any failures before continuing.

Step 8: clean up old tool artifacts

Once everything is migrated and stable:

  • Delete old Terraform state files or workspaces
  • Remove Helm release history if you used --keep-history
  • Archive old Kustomize overlays
  • Update CI/CD pipelines to stop running old provisioning

step 9: consider relaxing policies

After running stable for a while, you might want to adjust policies:

deployments:
  - id: app
    conflictPolicy: Stuck      # back to Stuck for safety
    deletionPolicy: Delete     # now safe to auto-cleanup
Enter fullscreen mode Exit fullscreen mode

Switching deletionPolicy back to Delete means when a node is deactivated, resources get cleaned up automatically. Only do this once you trust the system.

Keep conflictPolicy: Stuck for ongoing safety. Force was just for the migration.

troubleshooting common issues

Resource names don't match

If you see Lynq creating new resources instead of conflict events, your nameTemplate isn't producing the right names. Check the LynqNode spec to see what names it's trying to create.

Stuck on unexpected fields

The conflict message shows which fields conflict. Common culprits:

  • replicas (managed by HPA)
  • annotations (added by other controllers)
  • labels (injected by admission webhooks)

Use ignoreFields in your resource definition to skip these during reconciliation.

Old tool still trying to manage

If Terraform or Helm is still running somewhere (CI pipeline, cron job), it might fight with Lynq for ownership. Make sure you've fully disabled the old automation before migration.

LynqNode stuck in progressing

Check events: kubectl describe lynqnode <name>. Usually it's a dependency waiting for readiness or a template rendering error.

Rollback plan

If something goes wrong:

  1. Since you used deletionPolicy: Retain, resources are safe
  2. Delete the LynqNode: kubectl delete lynqnode <name>
  3. Resources stay in cluster, just unmanaged
  4. Re-import into Terraform: terraform import ...
  5. Or re-deploy with Helm: helm upgrade --install ...

The retain policy gives you this escape hatch. Use it.

Wrapping up

Migrating to database-driven automation doesn't have to be scary. The key is:

  1. Match existing resource names exactly
  2. Use Stuck policy to verify targeting before forcing
  3. Use Retain policy as a safety net throughout
  4. Migrate incrementally, not all at once
  5. Keep your old tool's state around until you're confident

Take your time. There's no rush. The resources aren't going anywhere.

Questions? Drop them in the comments or open an issue on GitHub.

Top comments (0)