DEV Community: Chris Psaltis

How to move files between Git repos and preserve history

Chris Psaltis — Tue, 06 Oct 2020 07:48:54 +0000

While working on a multitude of open source projects I faced an interesting Git puzzle. How can you move a file between Git repos?

The easy solution is to just forget about Git history. If you like to preserve it, the situation is not very straightforward. Below we will explain the problem, your options and how to apply them.

The problem

Let’s assume you work on forkX, which is a fork of repo projectX. In forkX you are collaborating with your colleagues. At the same time, projectX is also moving forward. ForkX includes files with a long history of commits that are not in projectX. You would like to push some of those files to projectX.

Your current situation looks like this:

What do you do?

Option A: Delete history

The easiest option is to push the files without preserving history.

Based on the example in the graph above, add relevant remotes and fetch their current state:

git remote add projectX git@github.com:someOrg/projectX git remote add forkX git@github.com:myOrg/forkX git fetch --all

Now, move to a clean branch that is identical to projectX/master.

git checkout projectX/master git checkout -b cleanBranch

Then, add your new files:

git checkout forkX/newFeature -- file_1 git add file_1

You can repeat for more files and commit:

git commit -m “Add new feature”

You are now ready to open a pull request. If you have the proper rights, you can just push with:

git push -u projectX

Other than the “Add new feature” commit, the pushed files will have no history upstream. The person who pushed will appear as the sole contributor.

This option is simple but will not work well when many people collaborate on the same files. You will end up deleting all contributors and every commit message along the way.

Option B: Preserve history

In order to preserve history the situation is more complicated.

Similarly to the previous option, add remotes, fetch their current state and move to a clean branch that is identical to projectX/master:

git remote add projectX git@github.com:someOrg/projectX git remote add forkX git@github.com:myOrg/forkX git fetch --all git checkout projectX/master git checkout -b cleanBranch

Then:

git merge --no-ff --no-commit forkX/newFeature

The command above will stop the merge with all the files staged and ready to be committed. For every file, besides the new ones, you do:

git reset filePath

Be careful NOT to reset everything and then stage the new files again. When the new files are the only ones staged then you commit with:

git commit -m "Add new feature"

Finally, delete the unstaged files with:

git stash && git stash drop git clean -f

You are now ready to open a pull request. If you have the proper rights, you can just push with:

git push -u projectX

Example

At Mist.io we rely on Apache Libcloud to interface with the APIs of many cloud providers. We maintain a fork of Libcloud to better control Mist’s dependencies and to develop new features before we push them upstream.

Until recently, we were maintaining a driver for vSphere only in our fork. The driver was big, complicated and introduced new dependencies so we refrained from pushing it upstream. When we felt confident with the code we decided to open a pull request.

The bulk of the new code was in a few files that didn’t exist upstream. However, the work on these files was done, over a long period of time, from several people in our team. For this reason, we wanted to preserve the history and we ended up using option B above.

Here is an example of how the same pull request looks like when pushed without history and with history.

Conclusion

In this post I went over how you can move files between Git repos. I showed two options; a simple one that deletes history and a complicated one that preserves it. I illustrated the two options with an example from Mist’s Libcloud fork and the upstream Libcloud repo.

I would love to hear your thoughts in the comments!

6 issues with vSphere's RESTful API

Chris Psaltis — Thu, 28 May 2020 15:31:44 +0000

This post is meant as a warning for users who are thinking to leverage vSphere’s RESTful API. More importantly, we hope that someone on vSphere’s team will read this and take action.

First, some brief history. vSphere's RESTful API, or Automation API as it is officially called, was introduced in v6.5. This initial version was lacking but in v6.7 it got better. Still, annoying issues persisted and going into v7.0 we were expecting to see some major improvements.

In Mist v4.3 we extended our support to vSphere v7.0. We were also planning to port our integration from the old SOAP API to the RESTful one. Unfortunately, the transition was impossible due to several issues.

In summary:

Calls for listings of resources return only a limited number of items and have no pagination.
You can’t use VM templates that reside in folders.
There is no way to resize disks.
UIDs are returned only for VMs but not for other resources e.g. clusters, hosts, datastores etc.
There is no information about hierarchical structures.
You need to juggle through requests to get the IP of a VM.

Keep on reading for more details.

1. Pagination

Automation API responses lack pagination and have a hard limit on the number of returned items. For example, "GET /rest/vcenter/vm" returns up to 4,000 VMs (reference). In v6.7-U3 the limit used to be 1,000.

What do you do if you have more than 4,000 VMs?

You first need to get all hosts with GET /rest/vcenter/host and then do a separate request for each one with GET /rest/vcenter/vm?filter.hosts={{host-id}}.

Keep in mind that GET /rest/vcenter/host has a hard limit of 2,500 hosts (reference). If you have more, you need an additional layer of nesting, e.g. get all datacenters, then loop for hosts and then loop for machines. Iterating like this, adds complexity and slows down code execution.

2. VM templates in folders

The Automation API supports only VM templates in Content Libraries (reference). It totally ignores templates in folders. No call returns them and there is no way to perform any action, e.g. move them to a Content Library.

This is a surprising omission, especially if you consider that VM templates are commonly used to deploy new machines. The only thing you can do is move your templates to a Content Library before using the Automation API.

3. Disk resize

There is no call to resize a disk. You can change the number of CPUs and the size of RAM, but not the disk. To add more disk space to a machine, your only option is to create a new disk and attach it. You are also unable to copy data between disks.

Bottom line, if you need disk resizing stick to the SOAP API.

4. UIDs and MoIDs

Starting in v7.0, the Automation API returns UIDs of VMs. For other types of resources you have to settle with Managed Object Identifiers (MoIDs). The problem is that MoIDs are not unique across vCenters.

This seems like a small fix, since the information is already there. We hope it will be available soon. Until then, be careful with IDs when you manage infrastructure on multiple vCenters.

5. Hierarchical structures

Several objects in vSphere have a hierarchical structure. For example, datacenter -> cluster, cluster -> host, host -> VM, folder -> VM etc.

Information about such structures is totally absent from API responses. To recreate it, you need to loop through all sorts of lists.

Let’s assume you want to find the folder in which a VM resides:

Get all folders with GET /rest/vcenter/folder. Notice that there is a limit of 1,000 returned items and the response includes only folder name, MoID and type (reference).
For each folder do GET /rest/vcenter/vm?filter.folders={folder-id}.
Then check if your VM is included in the response.

Such representations are useful very often and it’s hard to justify why they are not part of the API already.

6. VM IPs

If you want the get the IP of a machine you need to:

GET /rest/vcenter/vm/{vm-id} and get the NICs part. There you will find MAC addresses (reference).
GET /rest/appliance/networking/interfaces, for a list of all available interfaces. This is the only call that returns IP information (reference).
Search through the list of interfaces for the relevant MAC address and get the IP you need.

One would expect the IP to be available from the first call above (1) or at least offer a simpler way to cross reference items between calls.

Conclusion

In this post we went over the issues we had with vSphere’s Automation API. We also suggested some workarounds wherever possible.

Having to build and maintain integrations with more than 20 infrastructure platform APIs, we are accustomed to idiosyncrasies and annoying issues. Unfortunately, in the case of the Automation API the issues were too many to deal with.

Our hope is that future releases will solve these issues and the Automation API will become a first-class citizen in vSphere’s suite.

Comparing clouds: Billing for stopped machines

Chris Psaltis — Thu, 05 Mar 2020 09:56:02 +0000

Public clouds have grown considerably in size, complexity and sheer number of features. This makes it hard to answer even simple questions, especially when you are trying to compare clouds.

At Mist.io we hear such questions daily so we decided to do something about it. This is the first in a series of posts that compare clouds on a number of practical issues.

One of the questions we hear very often is some variation of the following:

Does my cloud bill me for stopped machines, aka instances, linodes, droplets etc?

The reasoning behind this question is quite simple. If I stop a machine, it means I’m not using it so I assume my cloud will not bill me for it. After all, public clouds are all about elasticity. If this is the case, then I could save a lot of money by stopping machines when they are not needed.

Unfortunately, things are not very straightforward.

Let’s go over, in alphabetical order, what is happening.

Service	Bills for stopped machines?	Notes
Alibaba ECS	Yes (by default)	Instances are billed per second. You could avoid billing for stopped instances connected to a VPC and which don’t have local disks. User action is required for that. If you turn this feature on and stop an instance, you will be billed for any of the following that apply: a) attached block storage b) associated elastic IPs c) bandwidth d) images For more details, check the official documentation for PAYG pricing here and specifically for stopped instances here.
Amazon EC2	No	Linux instances are billed per second with 60 seconds minimum. All others are billed per hour. When you stop an instance, you will be billed for any of the following that apply: a) attached block storage b) associated elastic IPs For more details, check the official documentation here and “Billing and purchase options” in this FAQ.
Digital Ocean	Yes	Droplets are billed per hour. Check the relevant answers in their pricing FAQ.
Google Compute Engine	No	Instances are billed per second with 60 seconds minimum. Some premium images follow a different model. When you stop an instance, you will be billed for any of the following that apply: a) persistent storage attached b) local SSDs c) associated static IPs For more details, check the official documentation here.
IBM Cloud	No	Public Virtual Servers and billed per hour. IBM offers “Suspended Billing”. Servers after Nov 1st 2018 include suspended billing. Most servers created before this date don’t offer it. If suspended billing is available and you stop a server, then you will be charged for any of the following that apply: a) storage b) secondary public IP address For more details, check the official documentation here.
Linode	Yes	Linodes are billed per hour. Check the relevant answers in their pricing FAQ.
Microsoft Azure	Maybe	Virtual Machines are billed per second and for the full number of minutes the machine was running. The documentation specifically mentions that if a machine was running for 6min and 45sec you will be charged for 6min. If the machine status is “Stopped Deallocated”, you are not billed. If it is “Stopped” or “Stopped Allocated”, you are billed for allocated virtual cores but not for software licenses. Full details on virtual machine states are available here. In order to get to "Stopped Deallocated" state, you have to stop the machine from within Azure's management portal or over the API using a specific deallocation parameter. If you stop the machine from within the OS it will go into “Stop Allocated” state. If you manage to get to "Stopped Deallocated" state, please keep in mind that you are still billed for any of the following that apply: a) attached Premium (SSD based) disks b) Attached Standard (HDD based) disks c) In ARM deployment model, you are billed for static public IP address unless it is part of the first five in the region. Read more regarding IPs under the FAQ section at the bottom of this page. For even more details, check the FAQ at the bottom of this page. The URL ends with /linux but you will also find the same FAQ under /windows…
Vultr	Yes	Vultr cloud instances are billed per hour. Check the relevant answers in their pricing FAQ.

For easier reference, you can also view the table above in Google Docs.

The comparison includes only services that offer cloud machines. There are also a number of services that offer dedicated hosts and/or bare metals. We didn’t include such services above because they are inherently different and, as expected, they charge you regardless of machine state.

Also, please keep in mind that the comparison refers to pay-as-you-go (PAYG) pricing. Alibaba, Amazon, Google, IBM and Microsoft offer reserved and spot pricing as well. In the case of reserved pricing, you will be billed even if you don’t use your reserved capacity. In spot, stopping a machine will usually release it and return it to the pool. Billing stops at that point, but you can no longer use the machine. This happens in Amazon, Google and Azure. In Alibaba and IBM, stopping a spot will not release it, but you will continue to incur charges until they either claim it back or you release it yourself.

If things were not complicated enough, you also need to take special usage discounts into account. Such discounts are:

Alibaba subscriptions
Amazon saving plans
Google committed-use and sustained-use discount

In the case of Alibaba subscriptions, things are rather simple. When you buy a subscription you pay a discounted price upfront for the entire billing cycle. Changing the status of the machine won't save you anything.

With Amazon saving plans, things are simple too. You commit to certain usage over 1yr or 3yr term and get a discount. If you use it, you’re good. If you don’t use it, you still pay for it.

Google’s committed-use discounts are very similar to Amazon saving plans.

Google’s sustained-use discounts are more complicated. First of all, Google follows an approach which they call resource-based pricing. In this model, the base price of a machine is tied to the underlying resources it is using (vCPUs and memory). If during your billing cycle you continue to run the same total amount of resources, then you gradually earn a discount that’s increasing over time. This is the sustained-use discount. The discount is irrelevant to the actual machines you run, it ties only into the total amount of resources used. This discount doesn’t increase linearly over time. To understand it better we strongly recommend reading the documentation pages linked above.

Having said all of the above, let’s restate the initial question:

Will I save money if I stop my cloud machines when they are not in use?

The answer depends on a number of factors. To get to the bottom of this you need to:

Check if your service will charge you for stopped machines and how.
Check your reservations and long term commitments.
Don’t take spot into account.
If you are using Google Compute Engine, do the math for the sustained-use discount.

All these might sound disheartening, but you could potentially save a lot of money. Just to get a sense of ROI, one of our customers was recently able to reduce a 5-digit monthly bill for dev infrastructure by 50%. They did it by automatically tagging machines upon provisioning and then setting a schedule to stop them during off business hours.

Bottomline, the effort is well justified. Do your research and good luck!