Every day is an API day
If you practice the piano everyday, you want to record your play and share it with your teacher, but you may not want to store it in your mobile or PC.
That's where API comes in!
This "Piano Lesson" is an application that captures video and send the data to AWS S3 bucket via MuleSoft API with WebSocket connection.
And,
This application completely runs on MuleSoft runtime, yes, self-contained! 1 CloudHub worker works as a web server, api server, and database server. You don't need any other server!!
Resources
Demo@CloudHub
- https://mule-worker-skd-piano-lesson.us-e2.cloudhub.io:8082/login
- My apologies for the self-signed certificate. To use the demo, you have to ignore the certificate error and trust it temporarily on your browser. I don't have CA signed certificates or Dedicated Load-Balancer for this project...
- You can use this credentials for the demo:
- username: saki
- password: saki
- Be careful for recordings... since we share the account above, all users can see your recording. If you want to delete your recording, you have to override the recording for the day and piece by re-recording.
- Please, please go easy on the S3 data storage... this application limits the size of 1 recording, but... please be kind!
GitHub
- https://github.com/ssakoda/piano-lesson
- Setup guide and Operation Manual is on the GitHub README.md
Architecture and Design
Architecture
This application uses 2 services - "MuleSoft CloudHub" and "AWS S3". AWS S3 provides only the file storage. All other functions are provided from a MuleSoft CloudHub runtime.
MuleSoft
The MuleSoft runtime application mainly consists of 4 flow configuration files - web, api, database and data. web and api contain "HTTP Listener" to expose functions to user devices, and others are just sets of internal logics.
By providing all endpoints from the same hostname and port, we can safely use "cookies" (with "SameSite=Strict) to implement "session" function (even between "https://" access and "wss://" access!)
web flow provides SPA (Single-Page-Application) resources powered by React framework. This SPA consists of only 2 resources (and favicon.ico) - HTML file and Javascript file. Since it's just 2 files, MuleSoft runtime can responses to the specific GET call by sending out the content of each HTML file and Javascript file by using File component and HTTP Listener.
AWS S3
AWS S3 provides 2 useful functions - "Multipart Upload" and "Object Presigned URL". "Multipart Upload" is very useful for this kind of applications, because the application itself doesn't have to buffer more than 5MB. The application just have to upload minimum 5MB chunks and keep ETags as the return of the upload until the end of the stream. "Object Presigned URL" is to download the stored data file to play the video on the application.
These can be called from client application side, but that causes sever security risks - exposing AWS credentials in the HTMT/js. That's why this client application calls MuleSoft API in the same runtime (hostname/port) with sharing the JWT token in the cookie and the API calls AWS S3 API with AWS credentials.
Considerations
Mobile Browser (iOS Safari)
This application doesn't work on iOS (Safari). Currently, iOS Safari doesn't support "Media Recorder" (By enabling Experimental Features "MediaRecorder", it seems work, but still it just supports "mp4" without buffering). Needs to handle "Media Stream" directly...
Database vs ObjectStore
This application uses ObjectStore as a database, but it just stores objects, has limitations like TTL, not suitable for production use as database against real database. For caching and temporary memory usages, Object Store is good, but not for transactional data storage.
Web Server...
Well... it's nice to have a modern web server (and CA signed certificate)
Design
Flows
web (skd-piano-lesson-web.xml)
This flow provides "SPA resources" and "WebSocket endpoint"
- SPA resources GET "/", "/index.html", "/index.js", "/login", "/login/login.html" and "/login/login.js" (and favicon.icon) responses the content of each file in src/main/resources. POST "/login" responses 302 with destination "/" only if username/password are qualified.
- WebSocket endpoint "/ws/data" with wss protocol acts as a WebSocket server with WebSocket component. According to https://docs.mulesoft.com/websockets-connector/1.0/websockets-connector-cloudhub, Shared Load Balancer (SLB) doesn't propagate ws/wss trafic (even though the hostname and port are the exact same as https!). Since I don't have Dedicated Load Balancer(DLB), I had to use "mule-worker" hostname and 8082 port for the demo (and that causes certification problem, sorry)
api (skd-piano-lesson.xml)
This flow provides server-side functions like database access, AWS S3 access. This uses API-kit and before the router, session (JWT in cookie) is always validated. It's good to have "web-and-API-server"!
database (skd-piano-lesson-db.xml)
This flow provides database access internally called by web and api flows via Flow Reference component. This function can be exposed to Mongo DB server or something like that, but this application seeks for the limit of "Object Store" capability (which is not recommended for the production use...)
data (skd-piano-lesson-data.xml)
This flow provides WebSocket data handling process called by web flow. The 1st message from WebSocket client is attributes like recording Id, and this process stores the mapping of "websocket key (sec-websocket-key header value)" and "recording id" in Object Store, so that the 2nd message (binary video data) can be linked with recording id via websocket key.
init (skd-piano-lesson-init.xml)
This flow has only "scheduler" to create the initial data (admin credentials). By executing this scheduler from Runtime Manager, initial data load can be achieved.
Implementation
Session and the validation
Since MuleSoft runtime is not a real web server, I had to implement a simple session function by myself, but it's not difficult as sound. The keys are "JWT" and "Set-Cookie header".
JWT (Json Web Token) is a compact URL-safe means of representing claims to be transferred between two parties. So I used this to encrypt user id with a secret string, and set the encrypted string in the client's browser cookie at the end of the successful login process.
MuleSoft provides encryption functions, but to handle the encryption in a clear way for me, I used Java implementation with "com.auth0.jwt.JWT" package (groupId = "com.auth0", artifactId = "java-jwt"). This is just easy like this:
To set the string to client's cookie, it just has to set the following string in "Set-Cookie" header of the response of the login request (after successful login process, it leads the user to "/", so "Location" header and status 302 is set in HTTP Listener response.
To validate the session, it just has to check the "cookie" request header and validate the session id string format, and validate the JWT.
This validation logic can be set not only API process, but also static resource access like "/index.html", and WebSocket server process, because they use exact same hostname and port number!
WebSocket
Originally, I used "socket.io" (https://socket.io/docs/v3/index.html) to implement the WebSocket client with Node.js, but surprisingly and unfortunately, it seems "NOT" compatible with MuleSoft WebSocket component.
What Socket.IO is not
Socket.IO is NOT a WebSocket implementation. Although Socket.IO indeed uses WebSocket as a transport when possible, it adds additional metadata to each packet. That is why a WebSocket client will not be able to successfully connect to a Socket.IO server, and a Socket.IO client will not be able to connect to a plain WebSocket server either.
https://socket.io/docs/v3/index.html
Ok... then, I had to use browser native WebSocket API. "socket.io" is useful, because it can send multiple kinds of data at once, like binary data and attributes as followings:
const socket = io('ws://localhost:3000');
socket.on('connect', () => {
// either with send()
socket.send('Hello!');
// or with emit() and custom event names
socket.emit('salutations', 'Hello!', { 'mr': 'john' }, Uint8Array.from([1, 2, 3, 4]));
});
// handle the event sent with socket.emit()
socket.on('greetings', (elem1, elem2, elem3) => {
console.log(elem1, elem2, elem3);
});
https://socket.io/docs/v3/index.html
But, native WebSocket API doesn't allow this kind of "send". I tried to use my own HTTP header, but native WebSocket API or MuleSoft WebSocket component doesn't allow to access to the header. I had to send "recording id" which is previously generated by API call "/api/start", and binary data with linking each other.
Then, I decided to send "recording id" as the 1st websocket message, then send binary data as the following messages. On the server (MuleSoft) side, to save the workload, checked the mediaType and stored mapping. The recording id is sent with mediaType "text/plain", binary data with "application/octet-stream". Once "text/plan" data is received, it stores the mapping of "atteributes.headers.sec-websocket-key" and payload (recording id). If "application/octet-stream", it can assume that the sec-websocket-key mapping is already in Object Store and get the recording id related.
This mapping is also used to buffer the binary data until 5MB.
Binary Data Handling
AWS S3 Multipart Upload is handy, but there is a limitation that "The minimum size of each part must be 5MB (except for the last part)". This "5MB" is sever for a MuleSoft runtime and browser... but Object Store can handle 10MB for a record.
This client application sends a binary data every 10 second, and the server (MuleSoft) side stores it in ObjectStore with sec-websocket-key as the key. At that time, the new binary data and stored binary data are concatenated, then stored again (if < 5MB).
The concatenation is handled by a custom Java class "BinaryUtils", because I couldn't find a way with DataWeave. "Binary" type is converted to "byte[]" in Java arguments/returns, so it just has to be concatenated by ByteArrayOutputStream, easy and simple.
If the mediaType is "application/java", sizeOf(data as Binary) works (because it is actually "byte[]"?), but not with "application/octet-stream" somehow.
RAML for API
#%RAML 1.0
title: skd-piano-lesson
types:
Error:
example:
{message : "Unknown error"}
properties:
message:
type: string
description: error message
User: !include schemas/user-dataType.raml
Piece: !include schemas/piece-dataType.raml
Recording: !include schemas/recording-dataType.raml
Playlist: !include schemas/playlist-dataType.raml
/me:
description: Handle login user information
get:
displayName: Get login user information
responses:
200:
body:
application/json:
type: User
401:
description: Unauthorized error
body:
application/json:
type: Error
/user:
post:
displayName: Upsert a User
description: if matched Id is found, this upserts the given record. if not, inserts. "password" has to be given as "Non-Encrypted" string.
body:
application/json:
type: User
responses:
200:
body:
application/json:
type: User
401:
description: Unauthorized error
body:
application/json:
type: Error
/{user_id}:
uriParameters:
user_id:
displayName: User Id
type: string
description: the Id of the User you query
get:
displayName: Get a User
description: Reterns a User if Id matched. "password" is given as "Encrypted" string.
responses:
200:
body:
application/json:
type: User
401:
description: Unauthorized error
body:
application/json:
type: Error
/pieces:
get:
displayName: Get all Pieces of the user
queryParameters:
inlist:
description: if true, this returns all "inlist=true" pieces of the user. if false, "inlist=false"
type: boolean
default: true
responses:
200:
body:
application/json:
type: Piece[]
401:
description: Unauthorized error
body:
application/json:
type: Error
/piece:
post:
displayName: Upsert a Piece
description: if matched Id is found, this upserts the given record. if not, inserts.
body:
application/json:
type: Piece
responses:
200:
body:
application/json:
type: Piece
401:
description: Unauthorized error
body:
application/json:
type: Error
/playlist:
get:
displayName: Get a playlist
queryParameters:
datestring:
description: target datestring (YYYY-MM-DD format
type: string
required: true
piece_id:
description: if specified, this returns a list with only the piece
type: string
required: false
responses:
200:
body:
application/json:
type: Playlist
401:
description: Unauthorized error
body:
application/json:
type: Error
/composers:
get:
displayName: Get a list of composers the user has
responses:
200:
body:
application/json:
type: array
items:
properties:
composer:
type: string
description: name of the composer
401:
description: Unauthorized error
body:
application/json:
type: Error
/recording:
/{id}:
uriParameters:
id:
type: string
displayName: Recording Id
description: Id of the recording you query
get:
displayName: Get a recording and the signed url
responses:
200:
body:
application/json:
properties:
recording:
displayName: the recording info
type: Recording
signedUrl:
displayName: Signed URL for AWS S3
type: string
401:
description: Unauthorized error
body:
application/json:
type: Error
/start:
post:
displayName: start recording
description: start recording by creating a recording record and preparing multi-part upload
body:
application/json:
properties:
piece_id:
displayName: Piece Id
description: Id of the piece for the recording
type: string
required: true
datestring:
displayName: Date String (YYYY-MM-DD)
description: Lesson date in YYYY-MM-DD format
type: string
required: true
responses:
200:
body:
application/json:
type: Recording
401:
description: Unauthorized error
body:
application/json:
type: Error
/stop:
post:
displayName: stop recording
description: stop recording by completing multi-part upload
body:
application/json:
properties:
recording_id:
displayName: Recording Id
description: Id of the recording to stop
type: string
required: true
responses:
200:
body:
application/json:
type: object
properties:
Location:
description: URL of the recording file
type: string
Bucket:
description: bucket name to be stored
type: string
Key:
description: Key of the recording file
type: string
ETag:
description: ETag for the upload
type: string
401:
description: Unauthorized error
body:
application/json:
type: Error
Latest comments (2)
Thanks for your submission @ssakoda !
Liked the way you hosted the solution !