In previous article, I explain briefly about Cosmos DB, Azure NoSQL service. In this article, I look into GCP.
GCP Databases
Same as Azure, GCP offers many database services, such as SQL, NoSQL, Cache, etc. See GCP Storage option for compare services. It also has good flow to find which service I should use depending on scenario.
SQL Databases
You can use Cloud SQL, which is cloud version of MySQL and PostgreSQL, and Cloiud Spanner. If you are into RDBMS, check out these services.
NoSQL
Datastore and Bigtable are GCP version of NoSQL service.
Warehouse
BigQuery is a warehouse solution for analytics purpose.
Bigtable and Cloud Datastore
Though both are marked as NoSQL services, BigTable is for more analytic purpose. So in this article, I focus on Datastore.
Refer to this document for detail about Datastore, but simply put, it's:
- Multi-document transactions
- Secondary & Composite indexes
- Automatically replicated across regions
- low-cost and secure
Sounds good.
Structure
Datastore is very different from Azure Cosmos DB where I simply store JSON objects. Though it is document base storage, it has own structure.
- Kind: It's equivalent to table.
- Entities: It's equivalent to rows. Kind has entities.
- Properties: An entity has properties to store field (column) values.
- Keys: It's primary key type of field.
- Index: It's not same index as RDBMS. DataStore needs index to run queries. Please watch the video in reference for more detail as it's a bit difficult to explain in one sentence.
There are many more things to learn about Datastore. I put useful links at the end.
SDK
Datastore offers SDK for multiple languages and REST endpoint. Therefore, if SDK is not available for your preferred language, you can still call REST endpoint.
SDK is available for C#, GO, Java, Node.js, PHP, Python and Ruby.
Let's code!
I was thinking about what is the best way to "compare" with Azure and GCP for NoSQL from developer experience. This maybe not fair to GCP, but I decided to modify the application which I used in the previous article to use Datastore to see how smooth I can convert to.
My strategies are:
- Keep the code as much as possible
- Try to generalize Datastore service
Okay, let's go.
1. Open the solution and manage NuGet. Remove "Microsoft.Azure.DocumentDB", and install "Google.Cloud.Datastore.V1".
2. Replace code in DocumentDBRepository.cs with following. Update projectId to your own.
namespace todo
{
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Threading.Tasks;
using Google.Cloud.Datastore.V1;
using System.Reflection;
public static class DocumentDBRepository<T> where T : class
{
private static string projectId = "cloud-compare-20180306";
private static DatastoreDb datastoreDb;
private static KeyFactory keyFactory;
public static async Task<T> GetItemAsync(string id)
{
try
{
var entity = await datastoreDb.LookupAsync(keyFactory.CreateKey(long.Parse(id)));
return ConvertToItem(entity);
}
catch (Exception e)
{
return default(T);
}
}
public static async Task<IEnumerable<T>> GetItemsAsync(Expression<Func<T, bool>> predicate)
{
Query query = new Query(typeof(T).Name)
{
Filter = Filter.And(Filter.Equal("Completed", false))
};
var entities = (await datastoreDb.RunQueryAsync(query)).Entities;
List<T> items = new List<T>();
foreach (var entity in entities)
{
items.Add(ConvertToItem(entity));
}
return items;
}
public static async Task<Key> CreateItemAsync(T item)
{
return await datastoreDb.InsertAsync(ConvertToEntity(item));
}
public static async Task<bool> UpdateItemAsync(string id, T item)
{
using (var transaction = await datastoreDb.BeginTransactionAsync())
{
Entity entity = transaction.Lookup(keyFactory.CreateKey(long.Parse(id)));
if (entity != null)
transaction.Update(ConvertToEntity(item, entity));
transaction.Commit();
return entity != null;
}
}
public static async Task DeleteItemAsync(string id)
{
await datastoreDb.DeleteAsync(keyFactory.CreateKey(long.Parse(id)));
}
public static void Initialize()
{
datastoreDb = DatastoreDb.Create(projectId);
keyFactory = datastoreDb.CreateKeyFactory(typeof(T).Name);
}
private static Entity ConvertToEntity(T item, Entity entity = null)
{
if (entity == null)
{
entity = new Entity()
{
Key = keyFactory.CreateIncompleteKey()
};
}
foreach (var property in item.GetType().GetRuntimeProperties())
{
if (property.GetValue(item) is string)
entity[property.Name] = new Value() { StringValue = property.GetValue(item).ToString() };
else if (property.GetValue(item) is bool)
entity[property.Name] = (bool)property.GetValue(item);
}
return entity;
}
private static T ConvertToItem(Entity entity)
{
var item = Activator.CreateInstance<T>();
foreach (var property in entity.Properties)
{
object value = null;
if (property.Value.ValueTypeCase == Value.ValueTypeOneofCase.StringValue)
value = property.Value.StringValue;
else if (property.Value.ValueTypeCase == Value.ValueTypeOneofCase.BooleanValue)
value = property.Value.BooleanValue;
item.GetType().GetProperty(property.Key).SetValue(item, value);
}
if (typeof(T) == typeof(todo.Models.Item))
item.GetType().GetProperty("Id").SetValue(item, entity.Key.Path.First().Id.ToString());
return item;
}
}
}
3. Open _Layout.cshtml and replace "Azure DocumentDB" to "GCP DataStore".
4. Now I need to authenticate to service. It's same as Storage Article, so I directly authenticate by running the following command locally. If you don't have gcloud yet, install it from here.
It launches the browser, so just authenticate.
gcloud auth application-default login
That's it. Simple enough? Well, not really actually...
What's different
Azure Cosmos DB and Datastore are very different as they store object in different ways.
Normal class vs Entity class
Azure Cosmos DB C# SDK let me use normal C# class to present document as it stores document as JSON, whereas GCP have me use "Entity" class, which I cannot even inherit from. This has several issues when I reuse code.
- Entity doesn't provide strongly typed experience for field.
- Entity has "Key" field, which I need to map to a field in my class.
- GCP Datastore C# SDK doesn't support LINQ and it has own Query class.
As I decided to re-use the code as much as possible, I tried to encapsulate everything inside DocumentDBRepository.cs
Initialize
Use ** typeof(T).Name ** to get type name as Entity name.
public static void Initialize()
{
datastoreDb = DatastoreDb.Create(projectId);
keyFactory = datastoreDb.CreateKeyFactory(typeof(T).Name);
}
Convert to Entity from normal class
I only implemented string and boolean type to simplify code, and use reflection. This affect performance.
private static Entity ConvertToEntity(T item, Entity entity = null)
{
if (entity == null)
{
entity = new Entity()
{
Key = keyFactory.CreateIncompleteKey()
};
}
foreach (var property in item.GetType().GetRuntimeProperties())
{
if (property.GetValue(item) is string)
entity[property.Name] = new Value() { StringValue = property.GetValue(item).ToString() };
else if (property.GetValue(item) is bool)
entity[property.Name] = (bool)property.GetValue(item);
}
return entity;
}
Convert from Entity to normal class
As I had to set Key value to a field in a class instance, I had to add Item class specific code, which kills generic purpose. I could change the Item class side but I wanted to keep other code intact.
private static T ConvertToItem(Entity entity)
{
var item = Activator.CreateInstance<T>();
foreach (var property in entity.Properties)
{
object value = null;
if (property.Value.ValueTypeCase == Value.ValueTypeOneofCase.StringValue)
value = property.Value.StringValue;
else if (property.Value.ValueTypeCase == Value.ValueTypeOneofCase.BooleanValue)
value = property.Value.BooleanValue;
item.GetType().GetProperty(property.Key).SetValue(item, value);
}
if (typeof(T) == typeof(todo.Models.Item))
item.GetType().GetProperty("Id").SetValue(item, entity.Key.Path.First().Id.ToString());
return item;
}
Handle Id field as Key field
Id field of Item is string type, whereas Id in Key is long. So I had to parse it for many operations.
public static async Task DeleteItemAsync(string id)
{
await datastoreDb.DeleteAsync(keyFactory.CreateKey(long.Parse(id)));
}
LINQ Query to Query class
I didn't want to manually convert everything, so I give up and simply hard code expected query.
public static async Task<IEnumerable<T>> GetItemsAsync(Expression<Func<T, bool>> predicate)
{
Query query = new Query(typeof(T).Name)
{
Filter = Filter.And(Filter.Equal("Completed", false))
};
var entities = (await datastoreDb.RunQueryAsync(query)).Entities;
List<T> items = new List<T>();
foreach (var entity in entities)
{
items.Add(ConvertToItem(entity));
}
return items;
}
Test it
Now code is completed, so try local debug first.
1. Hit F5 key to start debugging.
2. Add new item.
3. Go to Google Cloud Console and navigate to Datastore. You can see there is a record created.
4. Try update, and delete.
Deploy to Azure and GCP
Try deploy both platform. It works without any issue in GCP, but it fails in Azure with exactly same reason as Storage article, security.
Add authentication
At the moment, I just my own credential to test the application. It is okay while testing, but when I deploy the application, I need to enable authentication.
1. Run the following command to create service account. I gave project onwer, but you can limit it further depending on scenario.
serviceName=datastoreserviceaccount
projectId=<your project id>
gcloud iam service-accounts create ${serviceName}
gcloud projects add-iam-policy-binding ${projectId} --member "serviceAccount:${serviceName}@${projectId}.iam.gserviceaccount.com" --role "roles/owner"
2. Then run the following command to generate key.
gcloud iam service-accounts keys create keys.json --iam-account ${serviceName}@${projectId}.iam.gserviceaccount.com
3. Once the file is generated, copy it to local and add to Visual Studio 2017 project root.
4. Change the Initialize method to use the keys.json.
public static void Initialize()
{
var credential = GoogleCredential.FromFile("keys.json");
DatastoreClient.Create(new Channel(DatastoreClient.DefaultEndpoint.ToString(), credential.ToChannelCredentials()));
datastoreDb = DatastoreDb.Create(projectId);
keyFactory = datastoreDb.CreateKeyFactory(typeof(T).Name);
}
5. Also add using statements.
using Google.Apis.Auth.OAuth2;
using Grpc.Core;
using Grpc.Auth;
6. Deploy to Azure to test the application.
7. By the way, if you see "bower' is not recognized as an internal or external command" error while publishing, install it via npm.
npm install -g bower
In theory, this should work but it didn't. Maybe NuGet package loaded as part of GCP may cause the issue.
Explorer Tools
As a developer, I may need to check the data from outside of the application. There are several query tools are available. All the tools can be accessed via Google Cloud Console.
Query By Kind
I can add filters to quickly query data from Entities list.
Query By GQL
I can also write SQL like query.
Summary
Unlike Storage, I feel huge differences between Azure and GCP NoSQL services. I prefer Azure Cosmos DB from C# point of view as I can use normal class to define document and LINQ query and I don't have to mind too much about indexes. However, when I see how data stored, GCP may have better performance when I have tons of data.
Datastore has unique structure that means there is learning curb, for both coding and administering. For example, when I do simple enough query which select multiple fields require additional indexes. We have to carefully design data structure beforehand.
I look into Datastore deeper in the future as I cannot re-use my DocumentDB knowledge.
References
Cloud Datastore 101: Overview of Google's scalable NoSQL document database (Google Cloud Next '17)
Datastore Introduction
Datastore Query, Index and Transaction
Ken
Top comments (2)
You should check Firestore, which is the new generation datastore on GCP. It's more similar to Cosmos DB because you store data as JSON objects in it.
you are absolutely right :)