Turning Music Into Art — Building a Synesthesia Simulator with Gemini

sarthak bhardwaj — Sun, 14 Sep 2025 18:54:05 +0000

This is a submission for the Google AI Studio Multimodal Challenge

What I Built

I built the Synesthesia Simulator, an AI-powered applet designed to translate sound and imagery into a unified, cross-sensory artistic experience. It creatively simulates the neurological trait of synesthesia, allowing users to see music as color and hear pictures as melodies.

The applet provides a creative and exploratory space for users to discover novel connections between their senses. You can upload an audio file, an image file, or both, and the AI generates:

A Descriptive Scene – A vivid, artistic narrative describing the blended sensory experience.
Creative Prompts – Inspiring ideas for writing, art, or reflection based on the output.
A Generated Vision – A unique AI-generated image visually representing the fusion of sound and/or visuals.
Creative Chat – An interactive chat session with a creative AI assistant, primed with the context of your generated experience, to explore ideas further.

My goal was to create a tool that not only showcases advanced AI but also serves as a source of inspiration — particularly for creative and neurodiverse individuals who may naturally think in cross-sensory ways. It's not a medical tool, but a canvas for imagination.

Demo

Live Applet Link:

➡️ Launch the Synesthesia Simulator Here

Screenshots & Walkthrough:

Here’s the main interface where you can upload an audio file and an image:

After processing, the applet presents the AI's synesthetic interpretation alongside a newly generated piece of art. The app includes a built-in audio visualizer that reacts to your music, with customizable color schemes:

Other features and showing the experience with a context-aware creative AI assistant:

How I Used Google AI Studio

Google AI Studio and the Gemini API power this entire experience. I combined multiple models in a seamless pipeline to handle complex multimodal tasks:

Gemini 2.5 Flash (Multimodal Understanding):
- Core of the simulator.
- Handles system prompt + user prompt + audio file bytes + image file bytes all in one request.
- Outputs a structured JSON (descriptiveScene, creativePrompts, imageGenerationInstruction) for reliable integration into the UI.
Imagen 4.0 (Image Generation):
- Translates the imageGenerationInstruction from Gemini into tangible artwork.
- Creates visuals that embody the cross-sensory interpretation.
Gemini 2.5 Flash (Conversational AI):
- Powers the Creative Chat.
- A new chat session is initialized with descriptiveScene + creativePrompts as context.
- Turns the assistant into a creative partner, offering deeper exploration of the user’s generated experience.

Multimodal Features

The multimodal capabilities of Gemini are what make this applet possible:

Cross-Modal Understanding:
- Goes beyond analyzing audio and images separately.
- Interprets emotional tone of melodies, maps rhythms to textures, and links color palettes to musical patterns.
- Produces the descriptive scene that defines the synesthetic simulation.
Sense-Blending for Generation:
- Uses cross-modal insights to drive Imagen prompts.
- Example: “Abstract glowing waves of violet and silver flowing in rhythm with deep piano chords.”
- Generates true synthesis of sound + visual inputs.
Contextual Conversation:
- Creative Chat expands the experience.
- Users can ask: “What does the color red sound like in this song?” or “Tell me a story based on the third creative prompt.”
- The assistant responds with context-aware, imaginative answers.

✨ Thank you for checking out my project!

Submission by: @sarthak_bhardwaj_05aba55d

Prove Your Health Status, Not Your Identity: Building ZK-VCR on Midnight

sarthak bhardwaj — Mon, 08 Sep 2025 03:45:01 +0000

This is a submission for the Midnight Network "Privacy First" Challenge - Protect That Data prompt

What I Built

I built ZK-VCR (Verifiable Credential Oracle), a decentralized application that pioneers a new standard for privacy in on-chain transactions. It allows users to prove they meet specific health criteria (like having a low cardiovascular risk score) to a smart contract without ever revealing their underlying personal health information.

The project solves the "Leaky Bucket" problem of modern data privacy, where users are forced to hand over sensitive data to multiple services, risking exposure with every new interaction. ZK-VCR replaces this with an "Airlock" model, built on the philosophy of Privacy for the User, Transparency for the Algorithm, and Governance for the Source. A user's data never leaves their device; instead, a Zero-Knowledge proof is generated locally and sent to the chain for verification.

Demo

The complete source code and detailed documentation are available on GitHub:
Source Code & Complete Documentation

Here is a live recording of the ZK-VCR Command-Line Interface (CLI) in action, demonstrating the complete end-to-end flow from a clinic generating a credential to a user privately verifying it on-chain.

Watch the ZK-VCR Demo

How I Used Midnight's Technology

This project is built from the ground up using Midnight's core technology stack:

Compact Language: The entire on-chain logic, including the smart contract and all ZK circuits, is written in Compact. I used it to define the contract's state (like the owner and the trustedIssuers map) and to implement the complex, privacy-preserving logic inside the submitHealthProof circuit. Key features like persistentHash were used to create a ZK-friendly signature scheme.
MidnightJS SDK: The entire off-chain portion of the DApp—the user CLI, administrator panel, and issuer tool—is built in TypeScript and uses the MidnightJS SDK. This library was essential for all blockchain interactions, including:
- Deploying and finding contracts on the testnet.
- Managing wallets and private state.
- Constructing and submitting transactions to call the contract's circuits.
- Querying the public on-chain state to display it to the user.
Private Witnesses: The core of the DApp's privacy model relies on Compact's witness system. The user's sensitive VerifiableCredential and the administrator's ownerSecretKey are passed as private witnesses, meaning they are used in the ZK proof computation but never revealed on-chain.

Data Protection as a Core Feature

Data protection isn't just a feature of ZK-VCR; it is the central design principle.

The "Airlock" model ensures that the user's personal health information (PHI) never leaves their local machine. When a user wants to prove their eligibility for a service, the submitHealthProof ZK circuit is executed locally. The only artifact that is ever sent to the public blockchain is the anonymous ZK proof itself.

This proof mathematically demonstrates three things without revealing the underlying data:

The health data came from a trusted source (verified via a cryptographic signature).
The health data meets the publicly defined criteria (the "AI" model was run on it).
The proof is fresh and not being replayed (verified via a challenge-nonce).

The smart contract learns only a single binary fact: "An anonymous user has successfully proven they are low-risk." It learns nothing about their cholesterol, their blood pressure, or whether they smoke. This provides powerful, mathematically guaranteed privacy that is fundamentally superior to policy-based promises.

Set Up Instructions / Tutorial

The following is a complete, step-by-step guide to setting up the development environment and running the full ZK-VCR demo.

View the Full Tutorial on GitHub

Submission by: @sarthak_bhardwaj_05aba55d

DEV Community: sarthak bhardwaj

Turning Music Into Art — Building a Synesthesia Simulator with Gemini

What I Built

Demo

How I Used Google AI Studio

Multimodal Features

Prove Your Health Status, Not Your Identity: Building ZK-VCR on Midnight

What I Built

Demo

How I Used Midnight's Technology

Data Protection as a Core Feature

Set Up Instructions / Tutorial