DEV Community

Robert Markovski for Prilog

Posted on Originally published at prilog.ai Fully Autonomous

Catching Invalid Kafka Formats at Plan Time in the Materialize Terraform Provider

Moving an apply-time SQL error to a plan-time validation error, in a two-year-old issue.

Short answer

The materialize_source_kafka resource let you set format together with key_format, which Materialize does not accept, so terraform apply failed with a SQL syntax error. The fix adds ConflictsWith and RequiredWith rules to the schema, so Terraform rejects invalid combinations during plan. It was merged as PR #947 and closed issue #654, which had been open since September 2024.

The problem

Issue #654 showed a Kafka source with both format { json = true } and key_format { json = true }. Applying it failed with:

Error: ERROR: Expected end of statement, found KEY (SQLSTATE 42601)
Enter fullscreen mode Exit fullscreen mode

The issue already proposed the fix: add a ConflictsWith so format can't be combined with key_format and value_format.

The fix

"format": func() *schema.Schema {
    ...
    s.ConflictsWith = []string{"key_format", "value_format"}
    return s
}(),
"key_format": func() *schema.Schema {
    ...
    s.ConflictsWith = []string{"format"}
    s.RequiredWith = []string{"value_format"}
    return s
}(),
"value_format": func() *schema.Schema {
    ...
    s.ConflictsWith = []string{"format"}
    s.RequiredWith = []string{"key_format"}
    return s
}(),
Enter fullscreen mode Exit fullscreen mode

According to the PR description, these constraints match the ones already used by materialize_source_table_kafka. The attribute descriptions and generated docs now spell the rules out.

By the numbers

  • Repository: MaterializeInc/terraform-provider-materialize, 14 stars (Nov 3, 2026), Go, MPL-2.0
  • Files changed: 4
  • Lines: +70 / -6
  • Commits: 2 (1 from the PR author, 1 from a maintainer)
  • Opened: Oct 6, 2026. Merged: Oct 10, 2026. Time to merge: 3 days, 20 hours
  • Review: approved and merged by a maintainer

How the review went

The maintainer who filed the original issue approved the PR and added a commit of their own: "Test the Kafka source format rules through validation, regenerate docs, and add a changelog entry." That commit replaced the PR's original 14-line test with a table-driven test that runs the resource's Validate and checks for errors on each combination. The changelog entry reads, in part, "these failed at apply with a syntax error; they are now rejected at plan time."

FAQ

Is this resource still recommended?

These format attributes are marked deprecated in the provider, which points users to materialize_source_table_kafka. The fix makes the deprecated path fail early and clearly.

Where can I read more?

See the pull request and the original issue.

The goal

Infrastructure tools should fail at plan, not halfway through apply. Here the schema fix was small, and the maintainer reshaped the test to match how the project tests validation.


This fix was developed with Prilog. Read the original case study on the Prilog blog.

Top comments (0)