SAP Datasphere and Python

#sap #python #datasphere

This will be a short post, but given the challenges from SAP, I do believe that is important for those that works with SAP.

In recent years, SAP released the Business Central (SAP BC) and Datasphere, which follow a similar philosophy to Microsoft Fabric (a comparison I like to use given my Azure experience). The concept is to integrate all sources through built-in APIs or integrations within the SAP Datasphere, process the information, and then deliver that data into SAP Analytics Cloud.

The system itself is quite good when handling SAP-native data. In my experience, working with SAP products has always been fairly challenging, and the data is notoriously hard to extract. However, with this integrated approach, the entire process has become much smoother.

In the beginning, I relied mainly on views and minor processing within dataflows. But once I tried the Python script task, it was a game-changer!

Obviously, it is somewhat limited because you are only allowed to use the NumPy and Pandas libraries, alongside basic Python modules. Even so, this allowed me to move all my processing from SQL views directly into the dataflow. This enabled batching and parallel execution, making data loading significantly faster than using standard views.

But how to use Python Script from Datasphere?

The trick is:

Understand that this is the main function that will run the data. Once the data is there, you can use pandas dataframe logic with apply and lambda to add the processing in each row of it.

It is possible to add multiple functions inside this bigger function. Here this Forum from SAP explains really well the process.

Tips for creating and debugging:

1) It's really important to add the column that you are creating in the Python Script Task, and IN THE ORDER of the creation AFTER the incoming columns: