The Gaia Planetarium: A Full-Stack Embedded Python Project

If you are a regular on the developer community, you may have seen some recent posts about the InterSystems First Employee Programming competition. The challenge is simple — transform some Gaia Epoch Photometry data, calculate the percentage change of flux (between the maximum and minimum values), and output a list of the stars with a percentage change of greater than 100.

Now while this was a fun challenge, but I felt there was something missing: a lack of scope for creativity. After all, we are looking at stars! How can something as magical as the night sky be distilled down to a CSV files of IDs and flux values?

So I decided to do something different with my main entry, and visualise stars the way they should be viewed — in the night sky above us. Let's take a look at the Gaia planetarium:

Project Design

The first question was what information is needed to plot the stars in the nights sky? Now its easy to plot a point on earth - you just need a longitude and latitude. For stars, you also need two locating values, Right Ascension (RA) and Declination (Dec). Of course, it gets more complex if you want to know where they are in location to the earth, but that was solved later.

These values aren't in the Photometry datasets, instead being found in the Gaia source table, which can helpfully be queried using a Python library called astroquery. This made the decision to use Python a no-brainer.

So I decided upon the following stack:

- PyProd production
- Ingests Photometry data files
- Calculates output (and adds to csv file)
- Queries the Gaia Source table to get the location for each source ID.

- Flask Web Application (hosted using IRIS WSGI hosting):
- Query data from IRIS tables using Embedded Python
- Create REST Service to send data to the front-end

- Front-end UI with HTML/CSS/JS
 

Now, before going through the implementation in a bit more detail, I will detail one issue I found. The Gaia photometry data is heavy. Each zipped file is around 15.5MB, which doesn't sound like that much, until you realise there is >3000 of them. What's more, each of these files maps a small portion of the sky in immense detail, whereas I am much more interested in covering the sky with stars that could be visible to the naked-eye.

I therefore decided to add separate production components which can just query the Gaia source data, ordered by brightness to add the most visible stars to my planetarium. I've still included the photometry dataset and, because it is a production, its easy to throw more files into the watched directory to add them into map. The challenge results are available as an overlay, and may not be visible depending on whether the small portion of the sky that the photometry data maps is overhead.

I also added star data from a different dataset — Hipparcos, because many of the most visible stars overload the Gaia sensor, so are not available in the Gaia dataset. Including Hipparcos stars was important for viewing constellations to my planetarium. I also added constellation data from Stellarium to visualise the constellations in the sky.

IRIS Implementation

PyProd

I've recently written about PyProd for another example project, so I am going to skip the technical details in this case. One comment I will make though — it was really nice to be able to build a production in Python because I required a few steps which are super easy to do in Python, and not so easy in ObjectScript. The key example of this was querying the Gaia dataset directly with the astroquery.gaia library.

I also used this project as a moment to road test a new PyProd agent skill. This way, as soon as I start using PyProd, my agent can read the skill and see exact patterns of how it should be working with PyProd. A version of this skill is now available in the iris-agentic-dev skill library

The Web App: Flask

Its not super well known that IRIS can host WSGI applications, lets face it, I didn't even know what a WSGI application was when I first heard this! Web-Server Gateway Interface is a Python standard for running web applications. Some of the most popular Python web application frameworks, including Flask and Django, run on WSGI.

Flask is a lightweight framework for developing REST APIs. It has pretty simple syntax, where you define a function with a decorator to make it an REST endpoint. Combining this with Embedded Python makes it easy to get data from IRIS. For example, a simplified version of the endpoint which collects stars from the IRIS table is as follows:

from flask import Flask, request

import iris # Embedded python IRIS import 

app = Flask(name, static_folder="/home/irisowner/dev/src/skymap/static)


  
  
  Endpoint to access stars from the Database


@app.route("/api/stars", methods=["GET"])

def get_stars() 

    # SQL Query 

    query = """

        SELECT TOP 1000 SourceId, Ra, DecDeg, PhotGMeanMag 

        FROM "Gaia.SourceLocation"

        """

# Execute query 
rows = iris.sql.exec(query)

# Collect results
output = [] 
for row in rows: 
    output.append({
      "source_id": row[0],
        "ra"     : row[1],
        "dec"    : row[2],
        "mag"    : row[3]
    })

# Return results
return output

And we can also use this to activate a PyProd adapterless Business Service:

from intersystems_pyprod import director 


  
  
  POST endpoint


@app.route("/api/more-stars", method=["POST"])

def add_stars(): 

    # Get the info from the post request

    n_stars = int(request.args.get("n_stars"))

# Create Business Service
status, service = director.create_business_service("Gaia.StarCatalogService")

# Activate Business Service
service.process_input(n_stars)

Flask hosting

WSGI applications can be hosted through the Management Portal at System -> Security -> Applications:

But I wanted to do this programatically through Embedded Python. Using the Security.Applications.Create() function required me passing an IRIS array by reference into the function, which can be achieved using an iris.arrayref(<python dict>):

import iris
# Define settings 
props = iris.arrayref({
            'Type': 2,
            'NameSpace': 'USER',
            'WSGIAppLocation': '/home/irisowner/dev/src', # Path to flask project
            'WSGIAppName': 'skymap.server', # FolderName.FileName (without .py)
            'WSGICallable': 'app',
            'WSGIDebug': 0,
            'WSGIType': 1,
            'AutheEnabled': 64,
            'Enabled': 1,
            'Description': 'Gaia sky map Flask/WSGI application',
            "DispatchClass":"%SYS.Python.WSGI", # Needed for WSGI hosting
            "Path":"/home/irisowner/dev/src", # Path to flask project
            "WSGIDebug": 1, # Refresh when the code changes
            "MatchRoles": ":%All" # Security roles
        })

# Create Web App
sc = iris.Security.Applications.Create('/skymap', props)

An easy point to miss here if you normally create WSGI applications through the management portal: you need to set the DispatchClass to "%SYS.Python.WSGI".

Front-end

Unlike a pure Python framework like Streamlit, Flask include a Python front-end framework, instead relying on building a front-end with HTML/CSS/JavaScript (or some front-end framework).

I'm going to skip most of the details on this because it was largely a question of me describing clearly what I wanted, and getting some AI generated code as a result.

It is pure HTML/JS/CSS, which I personally am a big fan of. I know it has limits, but sometimes I think starting with a front-end framework overcomplicates things.

The Front end is hosted in the Flask app through a static route:

from flask import Flask, jsonify, request, send_from_directory

STATIC_DIR = os.path.join(os.path.dirname(__file__), "static")
app = Flask(__name__, static_folder=STATIC_DIR)

@app.get("/")
def index():
        return send_from_directory(STATIC_DIR, "index.html")

The only other thing to mention is that Cross Origin Resource Sharing is also handled from Flask

@app.after_request
def add_cors(response):
        response.headers["Access-Control-Allow-Origin"] = "*"
        return response

Conclusion

As I mentioned at the start of this article, this project started out as a competition entry for the Employee Programming Competition. However, it quickly spiralled into something else, once I realised the limitations of only using the Gaia Photometry Data. It has become a Python Full Stack Application, which integrates multiple data sources (Gaia, Hipparcus and files of constellations and star names).

To be clear though, it still does what was asked by the competition, and it even plots it as a bright yellow/orange/red patch (coloured by magnitude of flux change) of sky which is covered by the Photometry data. So if you've made it this far, please consider voting for my planetarium in the community vote!

I hope you've enjoyed reading about this Embedded Python project as much as I've enjoyed making it!