Azure Databricks: Fix “DataFrame Object Has No Attribute write”

#dataanalyst #pandas #pyspark #database

While writing data from an Azure Databricks notebook to SQL Server, I encountered the following error:

AttributeError: 'DataFrame' object has no attribute 'write'

The notebook was running on Azure Databricks Serverless Compute.

The error occurred because the DataFrame was created with pandas, but the code attempted to use the PySpark .write API.

The Problem

Suppose we have a Pandas DataFrame:

import pandas as pd

data = {
    "Product": [
        "Laptop", "Mouse", "Keyboard", "Monitor", "Headset",
        "Webcam", "USB Hub", "Laptop", "Mouse", "Monitor"
    ],
    "Region": [
        "North", "South", "East", "West", "North",
        "East", "South", "West", "North", "East"
    ],
    "Units_Sold": [15, 42, 30, 8, 25, 17, 60, 22, 38, 10],
    "Revenue": [
        22500, 1260, 2400, 3200, 1875,
        850, 1800, 33000, 1140, 4000
    ]
}

df = pd.DataFrame(data)

We then try to write the DataFrame to SQL Server:

df.write \
    .format("sqlserver") \
    .options(**SQL_OPTIONS) \
    .option("dbtable", "test_table") \
    .mode("overwrite") \
    .save()

The operation fails with:

Why This Error Happens

The variable df is a Pandas DataFrame, not a PySpark DataFrame.

You can confirm this by checking its type:

print(type(df))

The result will be similar to:

<class 'pandas.core.frame.DataFrame'>

Pandas and PySpark both use the name DataFrame, but they are different Python classes with different APIs.

A Pandas DataFrame provides writing methods such as:

df.to_csv()
df.to_excel()
df.to_parquet()
df.to_sql()

It does not provide the Spark .write property.

The following syntax belongs to the PySpark DataFrame API:

df.write.format(...).options(...).mode(...).save()

For a PySpark DataFrame, .write returns a DataFrameWriter, which is responsible for writing data to external storage systems and databases.

The Solution

Convert the Pandas DataFrame to a PySpark DataFrame before calling .write:

df = spark.createDataFrame(df)

After the conversion, the write operation works:

df.write \
    .format("sqlserver") \
    .options(**SQL_OPTIONS) \
    .option("dbtable", "test_table") \
    .mode("overwrite") \
    .save()

spark.createDataFrame() can create a PySpark DataFrame directly from a Pandas DataFrame. Spark converts the local Pandas data into a Spark DataFrame and infers the column schema when no schema is provided.