GraphQL Bulk Operation Example on Shopify API with PHP

In this post we will learn how to upload a list of products to our Shopify store by using their GraphQL API. It is a great tool for creating or updating multiple products on the same time if you have thousands of products on your store. Because with Shopify's REST API, you can only create one product per request and it might takes hours or even days to upload the products. So for the bigger stores that have thousands of products on their Shopify account, GraphQL API is a must to use tool.
But for the first time users of this API, it might be a little bit tricky to build the proper logic for uploading process. When I tried to do this for the first time, it took hours for me to understand the document of Shopify and then apply the logic to my code to upload my products. But here in this blog I will show you the code and explain every step of it.
First thing first, if you are here I suppose that you have a Shopify store :) and a bunch of product to upload on your store so I will not cover how to open a store on Shopify. Now as you have a Shopify store; open your admin dashboard page and go to Apps section.

In the apps section, go to Develop apps and click Create an app button.
After you create your app you will see your newly creaated app on your Apps list.

But to be able to use our newly created app we must first install it to our store and there are some configurations that we need to do for this.

As you see we need to get an access token and select our API access scope to install it. For this, go to Configure Admin API scopes.

As our application will be working on the backend side of our store, we will give all the access to our app. So we will select all the options from top to bottom;

After giving necessary access to our application, we can now install our app to Shopify store. If you have properly installed the app you will see your Admin API access token on your app's API credentials like this;

Once you reveal your access token, keep it in a safe place to be able to use it. Because Shopify shows this access token only once and after that you should copy and paste it in a secure file of your choice. Also keep your API key and API secret key too, as we will use them in our app.

After these steps now we can jump into our real job :D
Open up your favorite IDE and create a new PHP file to write our script.
We need to install the dependencies on our app before writing any code. (For this application, you need at least PHP 7.3 installed in your development environment. Also we use composer as our package manager.) Now type this on your terminal.

Shopify PHP library installation:

composer require shopify/shopify-api

After installing Shoppify API library we can use it on our app like this:

use Shopify\Clients\Graphql;

But to be able to use the classes from our composer libraries we need to do the autoloading with this code:

require __DIR__ . '/vendor/autoload.php';

After that, when we look at the documentation, it says every time you want to use your app, you should first set up your configurations. You can do that by calling the Shopify\Context::initialize method. Here is the example usage of the class that we can find in the documentation of Shopify.

use Shopify\Context;
use Shopify\Auth\FileSessionStorage;

Context::initialize(
    $_ENV['SHOPIFY_API_KEY'],
    $_ENV['SHOPIFY_API_SECRET'],
    $_ENV['SHOPIFY_APP_SCOPES'],
    $_ENV['SHOPIFY_APP_HOST_NAME'],
    new FileSessionStorage('/tmp/php_sessions'),
    '2021-10',
    true,
    false,
);

(To not having an error on this stage you might need to manually create the tmp folder inside of your vendor/shopify/shopifyApi/src/Auth directory.)
As you see now we are using our API credentials at this point. Since we don't want to use our API credentials directly on our script we will create an .env file on the main directory of our app. We will use a library called PHP dotenv for the easier usage of our environment variables. On your terminal, type this to install the library.

$ composer require vlucas/phpdotenv

Your .env file should looks like this:

SHOPIFY_ADMIN_API_ACCESS_TOKEN =  "YOUR ACCESS TOKEN HERE"
SHOPIFY_API_KEY = "YOUR API KEY HERE"
SHOPIFY_API_SECRET = "YOUR API SECRET HERE"
SHOPIFY_APP_HOST_NAME = "https://YOURSHOPNAME.myshopify.com/admin/api/2022-01/products.json"

SHOPIFY_APP_SCOPES = "read_analytics, write_assigned_fulfillment_orders, read_assigned_fulfillment_orders, write_customers, read_customers, write_discounts, read_discounts, write_draft_orders, read_draft_orders, write_files, read_files, write_fulfillments, read_fulfillments, read_gdpr_data_request, write_gift_cards, read_gift_cards, write_inventory, read_inventory, write_legal_policies, read_legal_policies, read_locations, write_marketing_events, read_marketing_events, write_merchant_managed_fulfillment_orders, read_merchant_managed_fulfillment_orders, read_online_store_navigation, write_online_store_pages, read_online_store_pages, write_order_edits, read_order_edits, write_orders, read_orders, write_payment_terms, read_payment_terms, write_price_rules, read_price_rules, write_product_listings, read_product_listings, write_products, read_products, write_reports, read_reports, write_resource_feedbacks, read_resource_feedbacks, write_script_tags, read_script_tags, write_shipping, read_shipping, write_locales, read_locales, read_shopify_payments_accounts, read_shopify_payments_bank_accounts, read_shopify_payments_disputes, read_shopify_payments_payouts, write_content, read_content, write_themes, read_themes, write_third_party_fulfillment_orders, read_third_party_fulfillment_orders, write_translations, read_translations"

And by the help of dotenv library we can just type

$_ENV['SHOPIFY_API_KEY']

if we need our API key.

By the way we took that SHOPIFY_APP_SCOPES array from this page of our Shopify admin dashboard:

Until now our code should looks like this:

<?php

require __DIR__ . '/vendor/autoload.php';

use Shopify\Clients\Graphql;


use Shopify\Context;
use Shopify\Auth\FileSessionStorage;
use Shopify\Clients\Graphql;


$dotenv = Dotenv\Dotenv::createImmutable(__DIR__);
$dotenv->safeLoad();

Context::initialize(
  $_ENV['SHOPIFY_API_KEY'],
  $_ENV['SHOPIFY_API_SECRET'],
  $_ENV['SHOPIFY_APP_SCOPES'],
  $_ENV['SHOPIFY_APP_HOST_NAME'],
  new FileSessionStorage('<...>/vendor/shopify/shopify-api/src/Auth/tmp/shopify_api_sessions'),
  '2021-10',
  true,
  false,
);

One thing you should be careful about until now is:

new FileSessionStorage('<...>/vendor/shopify/shopify-api/src/Auth/tmp/shopify_api_sessions'),

at this part you should enter the exact path of your tmp/sessions file.

After these first steps we will now start the bulk importing steps. This is the diagram that shows how the bulk importing works.

So first of all we will create a JSONL file and add our GraphQL variables to it. In the JSONL file, each line represents one product input. The GraphQL Admin API runs once on each line of the input file. One input should take up one line only, no matter how complex the input object structure is.
Here is the example JSONL file structure that we use in this project.

{"input":{"title":"Inventore excepturi ut.","productType":"provident","vendor":"Johns Group"}}
{"input":{"title":"Quod aspernatur.","productType":"corporis","vendor":"Vandervort-Rohan"}}
{"input":{"title":"Doloremque id.","productType":"ut","vendor":"Graham and Sons"}}
{"input":{"title":"Pariatur et velit.","productType":"dolores","vendor":"Koelpin, Hane and Steuber"}}
{"input":{"title":"Nisi laudantium.","productType":"voluptatem","vendor":"Keeling-Daniel"}}
{"input":{"title":"Exercitationem perferendis deleniti.","productType":"et","vendor":"Goldner, Nolan and Mertz"}}
{"input":{"title":"Cum architecto.","productType":"ipsa","vendor":"White Ltd"}}
{"input":{"title":"Mollitia at quo.","productType":"corporis","vendor":"Altenwerth Inc"}}
{"input":{"title":"Molestiae illum nihil.","productType":"dolorum","vendor":"Terry-Koch"}}
{"input":{"title":"Molestias dicta.","productType":"praesentium","vendor":"Swaniawski, Haley and Dooley"}}

You see we have 10 lines of input each representing one of the products. According to your product, input you can arrange your JSONL file.
Secondly, we need to upload our JSONL file to the Shopify. For that we need to first generate the upload URL and parameters.
We will use the stagedUploadsCreate mutation to generate the values that you need to authenticate the upload. The mutation returns an array of stagedMediaUploadTarget instances.
Also this will be a POST query to Shopify API. To be able to make a query to Shopify API we need a new graphQL instance too. And we need our access token that we kept in .env before. Here is the code for creating our instance:

$accessToken = $_ENV['SHOPIFY_ADMIN_API_ACCESS_TOKEN'];
$client = new Graphql("<YOUR-SHOP-NAME-HERE>.myshopify.com", $accessToken);

And as I like to put my queries in a variable, my query would looks like this:

$staged_upload_query = '
mutation {
  stagedUploadsCreate(input:{
    resource: BULK_MUTATION_VARIABLES,
    filename: "bulk_op_vars",
    mimeType: "text/jsonl",
    httpMethod: POST
  }){
    userErrors{
      field,
      message
    },
    stagedTargets{
      url,
      resourceUrl,
      parameters {
        name,
        value
      }
    }
  }
}';

This mutation will generate the values required to upload a JSONL file and be consumed by the bulkOperationRunMutation. So, I will take the response of that query and put it in a JSON file for see the results easily. Then we will use that variables in the next stage. Here is how I saved my response:

$response = $client->query(['query' => $staged_upload_que]);
$graphql_variables = $response->getBody()->getContents();
file_put_contents('graphql_variebles.json', $graphql_variables);

The response should look like this:

{
  "data": {
    "stagedUploadsCreate": {
      "userErrors": [],
      "stagedTargets": [
        {
          "url": "https://shopify.s3.amazonaws.com",
          "resourceUrl": null,
          "parameters": [
            {
              "name": "key",
              "value": "tmp/61275799751/bulk/893d0275-8da9-4308-b0b8-133338f97812/products.jsonl"
            },
            { "name": "Content-Type", "value": "text/jsonl" },
            { "name": "success_action_status", "value": "201" },
            { "name": "acl", "value": "private" },
            {
              "name": "policy",
              "value": "eyJleHBpcmF0aW9uIjoiMjAyMi0wMi0yNFQwOToyMjo0NVoiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJzaG9waWZ5In0sWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMSwyMDk3MTUyMF0seyJrZXkiOiJ0bXAvNjEyNzU3OTk3NTEvYnVsay84OTNkMDI3NS04ZGE5LTQzMDgtYjBiOC0xMzMzMzhmOTc4MTIvcHJvZHVjdHMuanNvbmwifSx7IkNvbnRlbnQtVHlwZSI6InRleHQvanNvbmwifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMSJ9LHsiYWNsIjoicHJpdmF0ZSJ9LHsieC1hbXotY3JlZGVudGlhbCI6IkFLSUFKWU01NTVLVllFV0dKREtRLzIwMjIwMjI0L3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QifSx7IngtYW16LWFsZ29yaXRobSI6IkFXUzQtSE1BQy1TSEEyNTYifSx7IngtYW16LWRhdGUiOiIyMDIyMDIyNFQwODIyNDVaIn1dfQ=="
            },
            {
              "name": "x-amz-credential",
              "value": "AKIAJYM555KVYEWGJDKQ/20220224/us-east-1/s3/aws4_request"
            },
            { "name": "x-amz-algorithm", "value": "AWS4-HMAC-SHA256" },
            { "name": "x-amz-date", "value": "20220224T082245Z" },
            {
              "name": "x-amz-signature",
              "value": "b48e6dc6a00d867d418151672670bbfac6b178953c45b4775518b9951d1fc52d"
            }
          ]
        }
      ]
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 11,
      "actualQueryCost": 11,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 989,
        "restoreRate": 50.0
      }
    }
  }
}

Now after having these variables, we will send a POST request to API with them and at the and of our request we attach our JSONL file that we prepared before. According to the documentation, the request should look like this:

curl --location --request POST 'https://shopify.s3.amazonaws.com' \
--form 'key="tmp/21759409/bulk/89e620e1-0252-43b0-8f3b-3b7075ba4a23/bulk_op_vars"' \
--form 'x-amz-credential="AKIAJYM555KVYEWGJDKQ/20210128/us-east-1/s3/aws4_request"' \
--form 'x-amz-algorithm="AWS4-HMAC-SHA256"' \
--form 'x-amz-date="20210128T190546Z"' \
--form 'x-amz-signature="5d063aac44a108f2e38b8294ca0e82858e6f44baf835eb81c17d37b9338b5153"' \
--form 'policy="eyJleHBpcmF0aW9uIjoiMjAyMS0wMS0yOFQyMDowNTo0NloiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJzaG9waWZ5In0sWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsMSwyMDk3MTUyMF0seyJrZXkiOiJ0bXAvMjE3NTk0MDkvYnVsay84OWU2MjBlMS0wMjUyLTQzYjAtOGYzYi0zYjcwNzViYTRhMjMvYnVsa19vcF92YXJzIn0seyJDb250ZW50LVR5cGUiOiJ0ZXh0L2pzb25sIn0seyJzdWNjZXNzX2FjdGlvbl9zdGF0dXMiOiIyMDEifSx7ImFjbCI6InByaXZhdGUifSx7IngtYW16LWNyZWRlbnRpYWwiOiJBS0lBSllNNTU1S1ZZRVdHSkRLUS8yMDIxMDEyOC91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0seyJ4LWFtei1hbGdvcml0aG0iOiJBV1M0LUhNQUMtU0hBMjU2In0seyJ4LWFtei1kYXRlIjoiMjAyMTAxMjhUMTkwNTQ2WiJ9XX0="' \
--form 'acl="private"' \
--form 'Content-Type="text/jsonl"' \
--form 'success_action_status="201"' \
--form 'file=@"/Users/username/Documents/bulk_mutation_tests/products_long.jsonl"'

But as we are using PHP, I changed this CURL request to proper PHP code. So after converting it, here is my code for this POST request:

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $curl_opt_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
$post = array(
  'key' => $curl_key,
  'x-amz-credential' => $curl_x_amz_credentials,
  'x-amz-algorithm' => $curl_x_amz_algorithm,
  'x-amz-date' => $curl_x_amz_date,
  'x-amz-signature' => $curl_x_amz_signature,
  'policy' => $curl_policy,
  'acl' => 'private',
  'Content-Type' => 'text/jsonl',
  'success_action_status' => '201',
  'file' => new \CURLFile('C:\Users\Fatih\Desktop\halukrugs\products.jsonl')
);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);

$result = curl_exec($ch);
if (curl_errno($ch)) {
  echo 'Error:' . curl_error($ch);
}
curl_close($ch);

At this part you will only change the exact path of your products.jsonl file.
And I will get the variables from the graphQLvariables.JSON file that we just created in the previous step. Here is how I get them:

$curl_opt_url = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->url;
$curl_key = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->parameters[0]->value;
$curl_policy = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->parameters[4]->value;
$curl_x_amz_credentials = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->parameters[5]->value;
$curl_x_amz_algorithm = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->parameters[6]->value;
$curl_x_amz_date = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->parameters[7]->value;
$curl_x_amz_signature = json_decode($graphql_variables)->data->stagedUploadsCreate->stagedTargets[0]->parameters[8]->value;

But of course you should put these variables before your curl request.

This request will return an xml file and we need the KEY of that result. I handled this by converting the result into an array and then take the KEY as a string which I will use in the next step as stagedUploadPath parameter.

$arr_result = simplexml_load_string(
  $result,
);

Now we can step into most critical part of our program. We will start creating our products in a bulk operation. We will use productCreate method inside of our bulkOperationRunMutation method. And we will pass it to the latter one as a string. And just as before I will put this query into a variable. Here is my query looks like:

$product_create_query =
  'mutation {
    bulkOperationRunMutation(
    mutation: "mutation call($input: ProductInput!) { productCreate(input: $input) { product {title productType vendor} userErrors { message field } } }",
    stagedUploadPath: "' . (string)$arr_result->Key . '") {
    bulkOperation {
      id
      url
      status
    }
    userErrors {
      message
      field
    }
  }
}';

As you see this is actually two mutation but the productCreate is running inside the bulkOperationRunMutation process. Also be careful that we use the KEY from the previous step as stagedUploadPath parameter.

And for the results of this query, you can use this code:

$product_create_response = $client->query(["query" => $product_create_query]);

Now if this operation would work well, you should have a response like this:

{
  "data": {
    "bulkOperationRunMutation": {
      "bulkOperation": {
        "id": "gid://shopify/BulkOperation/206005076024",
        "url": null,
        "status": "CREATED"
      },
      "userErrors": []
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 10,
      "actualQueryCost": 10
    }
  }
}

If you see this result, that means you have successfully created your products in your Shopify store, Congratulations!!! :D :) You can check it in your products page of your Dashboard.

But there is still some work that you might want to do. If you are using graphQL API, you must have thousands of products to upload Shopify. That means this process might still take some time to finish. And at this time you might want to check the status of your bulk operation whether it is completed or still running. Don't worry, it is an easy task.
To be able to check the status of our bulk operation, we need the id of that operation:

$bulk_op_id = json_decode($product_create_response->getBody()->getContents())->data->bulkOperationRunMutation->bulkOperation->id;

And we have another mutation for checking the operation status; currentBulkOperation.
I use that query inside a function so that I can call it in a timer. Here is my function for checking the status of bulk op. :

function check_bulk_op_status($client)
{
  $op_finish_que = 'query {
 currentBulkOperation(type: MUTATION) {
    id
    status
    errorCode
    createdAt
    completedAt
    objectCount
    fileSize
    url
    partialDataUrl
 }
}';

  $op_finish_resp = $client->query(["query" => $op_finish_que]);
  $str_op_finish_resp = $op_finish_resp->getBody()->getContents();
  $status_op_finish_resp = json_decode($str_op_finish_resp)->data->currentBulkOperation->status;
  return $status_op_finish_resp;
}

And by using that function, I check the status of my operation every 3 seconds (you can do longer in a real life scenario since it will take some time to finish the process and there is no need to check the status every3 seconds) with these line of codes:

$bulk_op_status = check_bulk_op_status($client);

while ($bulk_op_status != 'COMPLETED') {
  $bulk_op_status = check_bulk_op_status($client);
  var_dump('bulk operation status: ' . $bulk_op_status);
  sleep(3);
}

This was the last step of bulk product uploading process. When you see "Completed" on your terminal, then your operation is done.

Here is the complete code of this program:
Bulk PRoduct Upload

And you can find the documentation on here:
Bulk import
GraphQL API
Shopify API

Lastly, I just cover the basics of product uploading for the first time users of Shopify Admin GraphQL API. I can't say I used all the best practices on my code but this was just a basic example of how we can use the GraphQL API of Shopify. Once you get the basics you can do more complex product upload tools by yourself. Hope this was a helpful article for those who are struggling to understand the Shopify's documentation. See you on the next articles.

Comments

[tronics]

This is a fantastic walkthough!! Great work, please continue doing these kind of posts they are very helpful and eye opening at this level of detail.

I managed to use your code to upload multiple video and images through all those steps. I had to also use fileCreate to make that file accessible.

Btw I'd suggest du parse that array for the parameters like this instead of relying on the order (which may change):
$parameters=array();
foreach($item['parameters'] as $key => $val) {
$parameters[$val['name']]=$val['value'];
}
$post = $parameters;
$post['file'] = new \CURLFile($filenamepath,$mimeType,$filename);

Also by doing that it works with all types of media. Eg. images and videos have a different parameter set. Sometimes the CDN uses aws, othertimes Google.

Cheers

[Dimitris Papadopoulos]

Hi that was a great example of using Bulk Operations with PHP. I would like to ask how we can update variant inventory quantities using your tutorial. I managed to update only the product fields but variants seems a little bit tricky.