DEV Community: Edgar Moran

Faster Mule deployments using Gitlab cache

Edgar Moran — Thu, 31 Aug 2023 22:21:17 +0000

Today I was curious about how we can make our deployments faster using CI processes, we have multiple platforms to handle the CI deployments for example GitHub, GitLab, Bitbucket CircleCI, TravisCI etc.. In this case I’m using GitLab.

I created one application in MulesSoft with one simple scheduler and a logger, really I just want to test the deployment

the only couple important items to consider is to add the .gitlab-ci.yml file and to setup your build tag in your pom.xml file. Lets see how our .gitlab-ci.yml looks like:

image: maven:3.6.1-jdk-8

variables: 
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .m2/repository

stages:
  - build 
  - test
  - deploy-staging
  - deploy-production

build:
  stage: build
  script:
    - mvn  -U -V -e -B clean -DskipTests package
  only:
    - merge_requests

test:
  stage: test
  script:
    - mvn -U clean test
  only:
    - merge_requests
  artifacts:
    when: always
    reports:
      junit:
        - target/surefire-reports/TEST-*.xml  

deploy-staging:
  stage: deploy-staging
  script:
    - mvn -U -V -e -B clean -DskipTests deploy -DmuleDeploy
  rules:
    - if: '$CI_COMMIT_BRANCH == "staging"'

As we can see, I’m specifying these lines, with this, I tell Gitlab to cache the dependencies in the .m2 repository and the key there will allow to persist the dependencies in every branch.

variables: 
  MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .m2/repository

In my pom.xml this is the setup I have:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>com.mycompany</groupId>
 <artifactId>gitlab-cache-deployment</artifactId>
 <version>1.0.0</version>
 <packaging>mule-application</packaging>

 <name>gitlab-cache-deployment</name>

 <properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

  <app.runtime>4.4.0</app.runtime>
  <mule.maven.plugin.version>3.8.2</mule.maven.plugin.version>
 </properties>

 <build> 
  <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-clean-plugin</artifactId>
    <version>3.2.0</version>
   </plugin>
   <plugin>
    <groupId>org.mule.tools.maven</groupId>
    <artifactId>mule-maven-plugin</artifactId>
    <version>${mule.maven.plugin.version}</version>
    <extensions>true</extensions>
    <configuration>
     <classifier>mule-application</classifier>
     <cloudHubDeployment>
      <uri>${CLOUDHUB_URI}</uri>
      <muleVersion>4.4.0</muleVersion>
      <connectedAppClientId>${CLIENT_ID}</connectedAppClientId>
      <connectedAppClientSecret>${CLIENT_SECRET}</connectedAppClientSecret>
      <connectedAppGrantType>client_credentials</connectedAppGrantType>
      <environment>Sandbox</environment>
      <applicationName>gitlab-cache-deployment</applicationName>
      <workerType>Micro</workerType>
      <objectStoreV2>true</objectStoreV2>
     </cloudHubDeployment>
    </configuration>
   </plugin>
  </plugins>
 </build>

 <dependencies>
  <dependency>
   <groupId>org.mule.connectors</groupId>
   <artifactId>mule-http-connector</artifactId>
   <version>1.7.3</version>
   <classifier>mule-plugin</classifier>
  </dependency>
  <dependency>
   <groupId>org.mule.connectors</groupId>
   <artifactId>mule-sockets-connector</artifactId>
   <version>1.2.3</version>
   <classifier>mule-plugin</classifier>
  </dependency>
 </dependencies>

 <repositories>
  <repository>
   <id>anypoint-exchange-v3</id>
   <name>Anypoint Exchange</name>
   <url>https://maven.anypoint.mulesoft.com/api/v3/maven</url>
   <layout>default</layout>
  </repository>
  <repository>
   <id>mulesoft-releases</id>
   <name>MuleSoft Releases Repository</name>
   <url>https://repository.mulesoft.org/releases/</url>
   <layout>default</layout>
  </repository>
 </repositories>

 <pluginRepositories>
  <pluginRepository>
   <id>mulesoft-releases</id>
   <name>MuleSoft Releases Repository</name>
   <layout>default</layout>
   <url>https://repository.mulesoft.org/releases/</url>
   <snapshots>
    <enabled>false</enabled>
   </snapshots>
  </pluginRepository>
 </pluginRepositories>

</project>

Now I created couple repositories one for an application using the cache and a second one NO using the cache in the ci yaml file, this way we can validate and check performance between both apps. I created in both repos three branches (master, staging, mydevbranch). In order to verify performance, out pipeline has three stages

build: Only builds the project and verifies is succesful
test: will run the test (MUnit) in the pipeline
deploy: After a PR is approved and merged from Dev branch to Staging will deploy to Anypoint Platform.
The comparison
In the end using the cache improves in terms of minutes the time of running a build, test or deploy

No cache:

Build: took 1 minute, 37 seconds
Test: took 1 minute, 37 seconds
Deploy: Took 3 minutes 42 seconds

With cache

Build: took 32 seconds
Test: 38 seconds
Deploy: 3 minutes 42 seconds

As we can see there’s improvement on test and build while deployment seems to be the same, in the end a few minutes gained is better.

I will keep investigating if there are better ways to enhance the time of deployment, even some times the time is related to network, and the availability on the platform as well.

Hope this help you in your deployments!

Basic Google Big Query Operations with a Salesforce sync demo in MULE 4

Edgar Moran — Thu, 17 Feb 2022 20:21:56 +0000

If we think about data storage the first think it comes to our mind is a regular database, this can be any of the most popular ones like Mysql, SQL server, Postgres, Vertica etc, but I noticed no too many have interacted to one of the services Google provides with the same purpose Google Big Query. And maybe it is because of the pricing, but in the end many companies are moving to cloud services and this service seems to be a great fit for them.

In this post I would like to demonstrate in a few steps how we can make a sync job that allows us to describe a Salesforce instance and use a few objects to create a full schema of those objects (tables) into a Google Big Query Dataset. Then with the schema created we should be able to push some data into Bigquery from Salesforce and see it in our Google Cloud Console project.

In order to connect to Salesforce and Google Big Query, there are a few prerequisites we need:

Salesforce:

If you don't have a salesforce instance, you can create a developer one here,
From Salesforce side you will need username, password and security token (you can follow this process to get it)
A developer instance contains a few records, but if you need to have some more data, this will help the process to sync that information over.

GCP (Google Cloud Platform)

You can sign up here for free. Google gives you $300 for 90 days to test the product (similar to Azure). Also if you already have a google account you can use it for this.

CREATING A NEW PROJECT IN GCP AND SETTING UP OUR SERVICE ACCOUNT KEY.

Once you sign up for you account on GCP, you should be able to click on New Project option and write a project name, in this example I choose mulesoft

Once a project is created we should be able to go to the menu in the left and we should be able to select IAM & Admin > Service Accounts option.

Now, we should be able to create our service account

"A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs. Typically, service accounts are used in scenarios such as: Running workloads on virtual machines"

At the top of the page you should be able to see the option to create it, then just you need to specify a Name and click on create and continue,

Next step is to set the permissions, so for this we need to select from the roles combo BigQuery Admin.

Once created, we should be able to select from the three dot menu on the right the option Manage Keys

Then we can create a new Key, in this case one as json should be enough. The key will get downloaded automatically in your computer (Please keep this json key somewhere you can use it later.)

DATASET IN BIG QUERY

Datasets are top-level containers that are used to organize and control access to your tables and views. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery.

From the left menu we can search for BigQuery and click on it.

That will take us to the Bigquery console, now we can click in the three dots menu and select Create dataset option.

Now we just need to set the name as salesforce and click on "Create Dataset"

SETTING UP OUR MULE APPLICATION.

Since this is a sync job, we don't need any API specification but totally can fit some scenarios where we have another application that needs to consume specific endpoints / operations.

Let's then open our Anypoint Studio app (In my case I'm using mac) and let's use the default template. For this we are going to create five flows:

Sync. This flow just triggers the process.
DescribeInstance. This flow will be in charge of calling the describe operation using the Salesforce connector and provide all objects information from the Salesforce instance, also will have a loop that will allow us to process the job for the objects we are going to use.
DescribeIndividualSalesforceObject. Allows to describe an specific Salesforce object, this will basically will capture the fields and field types (STRING, EMAIL, ID, REFERENCE etc.) and will be on charge to create a payload that BigQuery will recognize in order to get created in GBQ
BigQueryCreateTable. This flow only will be in charge of creating the table in BigQuery based on the Salesforce object name and the fields.
QuerySalesforceObject. This flow dynamically will query the Salesforce object and pull the data (For this we are limiting the output to 100 records but in a bigger scale it should be done on a batch process of course.)
InsertDataIntoBigQuery. This flow will push the data over into BigQuery only

Now let's grab our json key generated by google and copy the file under src/main/resources folder. The key will let us authenticate against our project and execute the operations

IMPORT THE GOOGLE BIG QUERY CONNECTOR.

From Exchange we can search "Big Query" and we should be able to see the connector listed

then we can just use the "Add to project" option and we should be able to see the operations in the Palette

SYNC FLOW

As I mentioned, this is only in charge of triggering the whole application, so we only need one scheduler component and a flow reference to the DescribeInstance flow.

DESCRIBEINSTANCE

This flow will describe the whole Salesforce instance using the Describe Global operation. Next

steps on this is to use a Dataweave transform to filter to get only the Objects we are interested in, so in this case I'm only pulling three, Accounts, Contacts and a custom object called Project__c. I left in the transformation a few more attributes to only pull the objects that we are able to query.

%dw 2.0import try, fail from dw::Runtimeoutput application/java fun isDate(value: Any): Boolean = try(() -> value as Date).successfun getDate(value: Any): Date | Null | Any = ( if ( isDate(value) ) value as Date as String else value ) --- -(payload map (item,index) ->{ (item mapObject ((value, key, index) -> { (key):(getDate(value)) } ))})

view raw mapSalesforceReocrds hosted with ❤ by GitHub

Finally you need to loop over these three objects and there's a flow reference for this sample that will call the other flows to be able to continue the process.

DESCRIBEINDIVIDUALSALESFORCEOBJECT

The flow basically takes the name of the Salesforce Object and will allow to describe it, the connector only ask for the object name, then we have a pretty interesting DW Transformation

%dw 2.0input payload application/javaoutput application/javafun validateField(field) = if ( (field == "REFERENCE") or (field == "ID") or (field == "PICKLIST") or (field == "TEXTAREA") or (field == "ADDRESS")or (field == "EMAIL")or (field == "PHONE") or (field == "URL")) "STRING" else if ( (field == "DOUBLE") or (field == "CURRENCY") ) "FLOAT" else if ((field == "INT")) "INTEGER" else field --- -(payload.fields filter ($."type" != "LOCATION") map { fieldName : $.name, fieldType : validateField($."type")})

view raw Salesforce to Bigquery Fields Schema hosted with ❤ by GitHub

Salesforce data types are not 100% the same as BigQuery, so we need to make a little trick to be able to create the schema in BigQuery seamless as Salesforce so in this case I've created an small function to convert some fields like (ID, REFERENCE,TEXTAREA,PHONE, ADDRESS,PICKLIST, EMAIL) to be STRING, in this case the reference or values are not really anything else than a text, for (DOUBLE and CURRENCY) I'm using the value FLOAT and finally for INT fields are changed to be INTEGER

Finally because Location fields are a bit tricky and we are not able to make much with the API on them, I'm removing all location fields.

The output of this is the actual schema we will use to create the table in Google BigQuery.

BIGQUERYCREATETABLE

This flow allows us to create the table in BigQuery; we only need to specify Table, Dataset and Table Fields.

QUERYSALESFORCEOBJECT

This flow basically query the Object in Salesforce and then maps the data dynamically to prepare the payload for BigQuery.

The query basically comes from a variable "salesforceFields" same field we collected when we described the Object using this script

(payload.fields filter ($."type" != "LOCATION") map { fieldName : $.name}).fieldName joinBy ","

view raw salesforceFields hosted with ❤ by GitHub

And finally I'm limiting the result to only 100 records.

Next step is to map the Salesforce result data and map it dynamically using this script:

view raw mapSalesforceReocrds hosted with ❤ by GitHub

Thanks so much to Alexandra Martinez for the insights on the utilities for DW 2.0! (https://github.com/alexandramartinez/DataWeave-scripts/blob/main/utilities/utilities.dwl )

This last script basically maps the records and uses the key as field and the value, but the value needs to be replaced as Date in this case for the Strings that are date or date time. So I consider this the best script in this app.

INSERTDATAINTOBIGQUERY

This flow just inserts the data we prepared only, so basically we only need to specify table id , dataset id and the Row Data

SETTING UP OUR MULE APPLICATION.

Now we should be able to run our application and see the new tables and the data over Google big query.

On GCP I can see the tables I selected created:

And if we open any of them we should look into the schema to verify all fields are there

Finally we should be able to query the table in the console or clic on the Preview option to check the data is there.

I think this is kind of a common request we get on the integration space and many tweaks can be implemented if we are thinking of big migrations or setting some jobs that eventually will require tables to be created automatically from Salesforce to GCP.

If you like to try it, I created this GitHub repository. I hope this was useful and I'm open to hear any enhancement / scenario.

Image recognition using Mulesoft and Salesforce

Edgar Moran — Mon, 16 Nov 2020 22:01:48 +0000

Mulesoft and Salesforce seem to be the right combination of technologies to be able to deliver projects robust and complex in short time. I would like to demonstrate how we can use both of them to recognize images produced from a mobile device and recognize a picture bringing more information and interesting data for kind of a real scenario.

So, how this is done? Well here are some of components I'm using for this project (I will go deep on each one):

Salesforce developer Org
Anypoint Platform (Sandbox) account
Mulesoft mule-aws-recognition-system-api.
Mulesoft mule-aws-recognition-process-api

Salesforce developer Org

I have got a developer account for Salesforce from their developer site (developerforce.com). Salesforce in this case allows me to have:

Custom Objects (tables)
Custom Fields on each of those objects created
A way to expose a mobile application (previously known as Salesforce 1) available to install on IOS or Android devices
Visualforce pages, allowing to customize what we cant to show on a mobile app or browser
Apex Classes, custom apex code to handle data from a page or allowing to expose REST services from a custom apex (java style) definition.

So here we have the design:

Standard Object (Content Version and ContentDocumentLink). Allows to store the actual binary file in Salesforce
Custom Object (Hackathon Image). Allows to have a record to link the photo taken
Custom Object (Image Label). Stores the image information labels and how accurate is the image with the label from AWS.

here comes the fun part..

Visualforce page allowing to show an UI to take the picture:

Apex Controller. Allows to get all information from the picture and Create the Content Version and Content Link record related to the Hackathon image.
Apex Controller REST . Exposes the mentions endpoint allowing to trigger a push notification in the mobile device.

Where I can get this code from? https://github.com/emoran/sfdc-mulesoft-hackathon-2020.git

Now a basic flow:

Mulesoft mule-aws-recognition-system-api.

Initially this system api was for AWS only but because the time and resources I also included here one of the pieces I need to complete this exercise.

As I mentioned this system API allows to process a Base64 image and send it to Amazon Rekognition API, the result of this call is to be able to retrieve the labels generated from this call.

This same application contains the logic to pull a few tweets using a parameter based on hashtags.

#%RAML 1.0
title: mule-aws-recognition-system-api

/image:
  post:
    body:
      application/json:

    responses:
      200:
        body:
          application/json:

/twitter:
  /tweets:
    get:
      queryParameters:
        q:
          description: "Parameters to filter by hashtag"
      responses:
        200:
          body:
            application/json:

To process the image basically I used the AWS Java SDK to use the API my flow looks like this:

In the other hand for the Tweets we have a different endpoint which receives only the GET request and we return all tweets based on the hashtags provided.

Here how the flow looks like:

As you can see this is just a pretty simple HTTP Request to the Twitter API, It's not included in the process API as we are not using a connector to extract the logic of this request.

You can get the code of the system API from here: https://github.com/emoran/mule-aws-recognition-system-api.git

Mulesoft mule-aws-recognition-process-api

At this point in the process api, now we are really doing more things and connecting the dots. I will try to explain step by step what happens.

The process API has this RAML:

#%RAML 1.0
title: mule-aws-recognition-process-api


/image:
  post:
    body:
      application/json:
    responses:
      200:
        body:
          application/json:
            example:

/sfdc:
  /images:
    get:
      responses:
        200:
          body:
            application/json:
  /contentVersion:
    get:
      queryParameters:
        id:
          description: imageId
          type: string
/tweets:
  get:
    responses:
      200:
        body:
          application/json:

After the mobile application saves the picture we took with our device, Salesforce calls the /images endpoint we exposed in Mulesoft, basically it passes three params imageRecordId (Hackathon Image), contentVersionId (Id of the actual file in Saleforce) and contentDocumentLinkId (Link document to the picture.)
Mulesoft gets the parameters, then using the Salesforce connector we make a query to Content Version and we download the file (actual image in Base64), then we call the system API to passing the image and wait for the bunch of labels that AWS recognized

Once AWS responded we also create the labels in Salesforce (Image Labels) for the uploaded image as records and lastly we call the REST Service we exposed in Salesforce in order to notify the person the image has been processed and now it has labels created.

It was really interesting to check how to call that REST service from the connector, since on older versions of the connector we were able to connect getting the session ID and use the REST endpoint directly. In Mule 4 we are not able to do so, in this case we use the connector capabilities to do it

Now in the last part as user you can use your device to see the labels created per record, but also I created a feature on this process API. I've created a page served on Mulesoft to show the information we saved!

How I did do it?, in the same process API I placed a new configuration file named "portal", a flow that contains a "Load Static Resource" that serves a page stored in a folder named "web" on src/main/resources

Now the main page contains a script that uses JQuery in order to show the information of images and tweets.

This is how the render page looks like:

Basically the page tells you the labels we got from AWS, shows the image we took the picture and the tweets related to the generated labels.

you can get this code from here https://github.com/emoran/mule-aws-recognition-process-api.git

Watch the video:
http://www.youtube.com/watch?v=GWKP4U0o2Ng