DEV Community: Ash Moosa

Using Apache Solr in Production — Bitbucket

Ash Moosa — Thu, 22 Aug 2019 09:13:54 +0000

Using Apache Solr in Production — Bitbucket

This is a guest post by Pulkit Kedia, a backend engineer at Womaniya.

Solr is a search engine built on top of Apache Lucene. Apache Lucene uses an inverted index to store documents(data) and gives you search and indexing functionality via a Java API. However, to use features like full text you would need to write code in Java.

Solr is a more advanced version of Lucene’s search. It offers more functionality and is designed for scalability. Solr comes loaded with features like Pagination, sorting, faceting, auto-suggest, spell check etc. Also, Solr uses a trie structure for numeric and date data types e.g. there is normal int field and another tint field which signifies the trie int field.

Solr is really fast for text searching/analyzing and credit goes to its inverted index structure. If your application requires extensive text searching, Solr is a good choice. Several companies like Netflix, Verizon, AT&T, and Qualcomm use Solr as their search engine. Even Amazon Cloudsearch which is a search engine service by AWS uses Solr internally.

This article provides a method to deploy Solr in production and deals with creating Solr collections. If you are just starting with Solr, you should start by building a Solr core. Core is a single node Solr server, with no shards and replicas, while collections consist of various shards and its replicas which are the cores.

Implementation

In a distributed search, a collection is a logical index across multiple servers. The part of each server that runs a collection is called a core. So in a non-distributed search, a core and a collection are the same because there is only one server.

In production, you need a collection to be implemented rather than a Solr core, because a core won’t be able to hold production data (unless you do vertical scaling). Apache Zookeeper helps create the connection across multiple servers.

There are two ways you can set this up:

Multiple solr servers and use Zookeeper on one of the servers
Zookeeper on a different server and all the other Solr servers connecting to it

We’ll go through the process of implementing using the second approach. The first approach is similar to the second one but the latter is a more scalable approach.

Installing Solr

Spawn up 3 servers and install Solr on 2 servers (note: you can spawn any number of solr servers — we use 3 in our example). To install Solr, you need to install Java first, then download the desired version and untar it.

Installation: wget http://archive.apache.org/dist/lucene/solr/8.1.0/solr-8.1.0.tgz

Untar: tar -zxvf solr-8.1.0.tgz

You can start Solr by going to the /home/ubuntu/solr-8.0.0 folder with bin/solr start or in the bin folder with ./solr start. This would start solr on port 8983, and you can test it in the browser.

Replicate the exact same steps to install Solr on your 2nd server.

Also remember to setup the list of IP’s and names for each in /etc/hosts

For example :

IPv4 Public IP-solr-node-1 solr-node-1 IPv4 Public IP-solr-node-2 solr-node-2 IPv4 Public IP-zookeeper-node zookeeper-node

Installing Zookeeper

Now the 3rd server would require only zookeeper to which you would push configsets.

Installation: wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

Untar : tar -zxvf zookeeper-3.4.9.tar.gz

If you like, you can add the path to zookeeper to the bashrc file.

Next, in the zookeeper-3.4.9 folder, there is a sample configuration file which comes with zookeeper -> zoo_sample.cfg. Copy this file in the path and rename it to zoo.cfg. The configuration file contains various parameters like “ dataDir ” which specifies the directory to store the snapshots of in-memory database and transaction logs, “ maxClientCnxns ” which limits the max number of client connections etc.

Open the zoo.cfg file and uncomment “ autopurge.snapRetainCount=3 ” and “ autopurge.purgeInterval=1 ” and edit the “ dataDir = data ”

Next start the zookeeper.

bin/zkServer.sh start

Creating A Configset

Configsets are basically the blueprint of the data to be stored. Configsets are stored at server/solr/configsets

You can create your own configset and use it to store your data. Change the managed-schema file content to customise the config.

You can modify the tag to denote the data fields to be stored in one document
you can define the type or create a new type by defining it with the tag.
the id field is compulsory so you cannot delete that

There are many other things you can do in Solr like dynamic fields, copy fields etc. Explaining each of them is beyond the scope of this blog but for more information, here is the official documentation.

Now that you’ve created a config and have chmod -R 777 config folder, push the config to the zookeeper.

bin/solr zk upconfig -n config\_folder\_name -d /solr-8.0.0/server/solr/configsets/config\_folder\_name/ -z zookeeper-node:2181

After pushing the config, start SolrCloud on each Solr servers. To install SolrCloud, refer to this documentation.

Connecting to Zookeeper

To connect to the zookeeper:

bin/solr start -cloud -s example/cloud/node1/solr/ -c -p 8983 -h solr-node-1 -z zookeeper-node:2181

Solr stores the inverted index at this location -> example/cloud/node1/solr/ , so you need to mention that path while connecting. Zookeeper will automatically distribute shards and replicas over the 2 solr servers. When you add some data, a hash would be generated and then it would in a particular shard. This is all handled by zookeeper.

To add data to the server you need to POST to the link http://:8983/solr//update?commit=true .

The IP can be of any server , the data automatically gets distributed among the shards.

To get data from your solr search http://:8983/solr/user/select?q=

Note: If you are using one of the Solr servers as a zookeeper, all the above steps are the same but replace zookeeper ip with that solr nodes ip and port to 9983 instead of 2181

Troubleshooting

Here are a couple common problems that may arise while setting up SolrCloud.

After you have created SolrCloud and are connecting to zookeeper, you may see an error like 8983 or 7574 is already in use.

:fuser -k 8983/tcp -

This would find the process and kill it

Another error you may see is that SolrCloud cannot find the newly created configset.

Solution: Do chmod 777 to the new configset. The more secure approach is to chown the folder to solr user.

Conclusion

Solr has a large community of experienced users and contributors and is more mature when compared to its competitors. Solr faces competition from Elasticsearch, which is open source and is also built on Apache Lucene. Elasticsearch is considered to be better at searching dynamic data such as log data while Solr handles static data better. In terms of scaling, while Elasticsearch has better in-built scalability features, with Zookeeper and SolrCloud, it’s easy to scale with Solr too.

Author bio : Pulit Kidia is a backend engineer with experience in cloud services, system design and creating scalable backend systems. He loves to learn and integrate new backend technologies. Get started, it’s free

Originally published at https://bitbucket.org on August 22, 2019.

Automate Coverage Reports in Pull Requests with Bitbucket, Jenkins and SonarCloud — Bitbucket

Ash Moosa — Tue, 16 Jul 2019 07:00:37 +0000

Automate Coverage Reports in Pull Requests with Bitbucket, Jenkins and SonarCloud — Bitbucket

At Instaclustr, we’ve experienced significant growth in our team sizes that has been great for increasing the scope and speed of our development. The flip-side to this benefit is that with the increased velocity of projects came increased pressure on approvers to take time from their own tasks, and provide quality feedback at a faster pace.

To address this, we overhauled our existing build systems to:

Automate static code analysis
Expose important metrics (such as test coverage, whether tests have passed); and
Expose it to reviewers within pull requests

Now, our review workflow is:

Developer creates a PR in Bitbucket, targeting the release branch
Jenkins sees the creation of the PR and starts our build-and-test pipeline beginning with unit and system tests. If successful, the pipeline progresses through to our end-to-end tests. At each stage, coverage results are forwarded to SonarCloud for analysis
When an approver views the PR, Bitbucket (via the SonarCloud widget) pulls in the code analysis results and provides relevant context

Approvers can immediately know:

Coverage of new code code in the PR
If there are any common code errors (e.g. not closing resources)
That style guidelines have been followed (e.g. how deep is the inheritance tree)

Best of all, relatively few changes were required to implement this!

1. Jenkins

We use Jenkins as our build system, so we created a multibranch pipeline job that uses the Bitbucket Branch Source Plugin to poll for any new or updated PRs targeting our release branch. The pipeline trigger can then be configured to scan every minute.

Once triggered, the job will run our test pipeline Jenkinsfile.

The relevant parts of our Jenkinsfile are:

mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent test --fail-at-end -DskipTests=false -am

mvn sonar:sonar --batch-mode --errors " + "-pl ${context.env.TEST\_MODULES} -am " + "-Dsonar.projectKey=${Constants.SONARCLOUD\_PROJECT\_KEY} " + "-Dsonar.organization=${Constants.SONARCLOUD\_ORGANISATION} " + "-Dsonar.verbose=true " + "-Dsonar.host.url=${Constants.SONARCLOUD\_URL} " + "-Dsonar.login=${context.env.SONARCLOUD\_TOKEN} " + "-Dsonar.pullrequest.branch=${context.env.BRANCH\_NAME} " + "-Dsonar.pullrequest.base=${Constants.RELEASES\_BRANCH} " + "-Dsonar.pullrequest.key=${context.env.CHANGE\_ID}

2. SonarCloud

Uploaded reports will automatically register a new PR in SonarCloud and can be explored through the SonarCloud Console to show inline views of code issues, test coverage trends and whether the PR meets customisable Quality Gates. It’s important to note that these metrics are calculated against new code introduced by the PR, so developers don’t have to sort through an analysis of the entire codebase.

3. BitBucket

At this point, we have Jenkins automatically testing PRs and SonarCloud providing analysis. To make it as easy as possible for approvers to see that information, we just need to enable the SonarCloud widget in our Bitbucket repository.

Now, whenever a PR is created, the widget will pull in test coverage, bugs and code smell metrics. Using the Bitbucket Jenkins plugin also means that our PRs will show a handy “build passed” status to let us know when a branch has is successfully passing all test cases.

Summary

For Instaclustr, this setup has:

Simplified the basic checks a reviewer would undertake by providing summary metrics for assessing the quality of a PR
Highlighted test cases that don’t work in a parallel setting (e.g. incorrect usage of singleton objects)
Highlighted poor test verification patterns (e.g. count pre- and post-test comparison rather than checking for presence of record)
Code coverage highlights when tests aren’t being run (e.g. names not matching Surefire inclusion patterns)
Ensures highlighting of simple code errors that can have significant impacts, such as not closing resources

Hopefully this has provided a quick guide to getting up and running with static code analysis; if you have any feedback, ideas or questions about this article, I would love to hear it!

Author bio: Alwyn Davis is a Senior Software Developer at Instaclustr where he has worked on multiple infrastructure and development projects, in addition to providing client-facing support and delivering consulting projects. He has focused on the development of Instaclustr’s client-facing management systems implementation of Cassanda, Spark, and Kafka deployment processes. He also has a background in technical consulting experience in search engine, database and CRM implementation, management and application development.

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Scaling your Bitbucket team? Upgrade your plan here

Originally published at https://bitbucket.org on July 16, 2019.

Software development with UML and modern Java

Ash Moosa — Wed, 05 Jun 2019 17:29:52 +0000

This post was written by Bitbucket user Aleksandar Radulović.

With technology evolving fast, there is a need to write and maintain software more efficiently, and better communicate with team members. As developers, we rarely get to think about these things as we rush to meet deadlines.

Current software development practices rarely include software modeling. Even when models are used, they are mostly used as a part of the documentation process and often seem more of a burden.

The purpose of this article is to describe a different approach to software development that puts visual modeling and code generation into the heart of the development process. Visual software models put emphasis on communication and internal software design rather than simply making things work.

I’ll describe how we use code generators to automate software development by using a UML model as a starting point for creating modern Java back-end applications using frameworks such as Spring, Spring Data and Hibernate.

In order to understand the potential of this approach, we need to consider different cornerstones of the development process and the impact this approach has on them:

Unified Modeling language (UML) — used to visualize software systems and flows
Modern Java — essentials of the modern Java ecosystem
Building the Software Product — faster prototyping and maintenance
Team dynamics — better communication and faster onboarding

Let’s briefly go through each of these.

Unified Modeling Language

The Unified Modeling Language (UML) is a standardized, visual language for modeling software. It was developed with an ambitious intention: to provide software teams a standard way to visualize the design of a system and to improve the team’s understanding of the domain of the problem they were solving. Using UML, one can visually model concepts, processes, state machines, interactions or use cases.

The approach we take in our day to day work is to use class diagrams for modeling domain concepts and relations between them and state machines for modeling process flows. We also document different model elements: classes, interfaces, attributes, etc. so that we can derive documentation from the model at any time, using different formats and structures: javadoc or Swagger, just to mention the two.

Here is an example of a UML class diagram.

Modern Java

Modern Java has a vibrant ecosystem. While it takes time to learn a programming language, adopting modern frameworks from that language’s ecosystem is an additional learning curve.

Declarative programming

The emergence of declarative software development practices has silently opened new ways for model-driven development. Unlike the imperative programming flows that are inconvenient to specify using modeling techniques, declarative programming constructs describe structural aspects of the software that can naturally be represented by the class diagrams.

Contemporary Java development heavily relies on declarative constructs: annotations most of all. For example, different frameworks, such as Hibernate and Jackson, use annotations to map object models to relational databases or to different export formats (JSON, XML, Protobuf, BSON, CSV). The Spring Framework, among many other things, brings great support for declarative development of RESTful endpoints and Spring Data introduces many essential constructs for abstracting data store access operations.

Like other types of programming, declarative programming does come at a cost — we introduce complexities of different frameworks and libraries into our applications. While these dependencies bring complexity to the project, developers must learn that they offer a return on developer productivity by letting them focus on high level objectives.

Code generators

Declarative programming allows for code generation. Instead of having to write Java annotations by hand, it is enough to mark a class as persistent in the model and let the code generation tool create Java Persistence API (JPA) annotations for you. Instead of having to write lines and lines of JPA annotations, which can be cumbersome at times, code generation can do the magic without letting you bother with the details. Code generation is either built in to the UML tool you’re using or may be available as a plugin — it’s usually a one click process to go from UML to code.

Here is a sample of the Java code generated from the UML model shown above.

Why use code generators? Code generators translate the language of design (UML) into the language of implementation (Java). It brings automation to our development process, reducing overall development complexity and simplifying maintenance. We can be truly focused on modeling application concepts and services, the core abstractions we are dealing with, while the code generator synchronizes the model with the codebase. Further, it promotes the usage of best practices and significantly impacts the quality as well as the uniformity of the codebase.

Software Product

When the model is completed, the code generator creates a complete starting project that reflects our design — so we can focus on implementing business logic. When it comes to software maintenance, you can change the design and let the generator propagate changes to your codebase. This process of working with a software model and using a code generator allows for rapid prototyping, easier software maintenance and gives you better documentation of your product.

The question that quickly arises when you start working with code generators is: how to synchronize changes that you introduce in the code with the model? Our answer to that question is simple: don’t do that. The model is a set of abstractions and it should be kept separate from implementation.

This one-way transformation is typically referred to as the “model first” approach because it clearly puts emphasis on modeling and not vice versa.

On the other hand, we still want to be able to modify the generated Java code. For that purpose, we rely on preserved sections within Java source files, that keep custom changes intact through multiple code generations.

Development Team Now Has a Visual Map

Team development and communication are often underestimated topics in the everyday hectic run towards achieving results.

Using code generation brings the UML model to the heart of the software development process. The UML model of the product becomes a visual map that evolves as the work progresses. Having this map, different team members can understand the software better and have focused discussions. Onboarding of new team members is now much faster: instead of reading lines and lines of code, they rely upon a live map that communicates backbone ideas without implementation specific details.

This visual software development technique changes the traditional responsibilities of the team members, promoting mutual understanding of the domain and improving team cohesion. When using model-driven development, the role of software developer comes closer to the role of a business analyst. On the other hand, a business analyst clearly understands how the software is being built and the relationships between domain concepts. Finally, QA engineers have a better understanding of the application, and all team members speak the same language.

Conclusion

While it’s possible to use code generation and modeling to automate parts of software development, we do not see this often in practice — either due to lack of awareness or lack of resources to invest in reviewing and researching new ways of working. If the ideas expressed in this article get you interested in model-driven development, there are several ways to go further.

There are multiple providers of low-code development solutions. Mendix is one of them and has a comprehensive guide to low code development.

The best open-source example of this category of products is JHipster, a project that has been embraced by thousands of developers worldwide. The JHipster core team managed to connect experts from different areas of software development to make this amazing application generator.

Our own endeavor is in extending StarUML, our preferred tool for software modeling, with a plugin for code generation — this is the plugin used in the example in this post.

Finally, no matter which tools and methodologies you use, software development is a people business and as such, it has many different sides that are difficult to measure and manage. Model-driven development cannot replace the lack of quality requirements, lack of empathy within the team, or lack of organizational culture in general. It complements agile development methodologies but does not replace them.

Author bio: Aleksandar Radulović is a software developer and architect who developed Rebel, a code generator plugin for StarUML. When he’s not developing software, he enjoys reading classics like Shakespeare or is dancing the tango. Connect with him on Linkedin.

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on June 5, 2019.

How Building An IDE Extension Changed The Way We Ship Code

Ash Moosa — Wed, 15 May 2019 21:53:29 +0000

When our team set out on the adventure of building the Atlassian for VS Code extension, our mission was simple: create an MVP to test if using Bitbucket Cloud and Jira Software Cloud features inside of VS Code would make a better developer experience.

To begin, we did what we all knew: scheduled planning meetings, had daily stand-ups, setup a Slack channel for all of the discussion that happens between meetings, tried to guess at release dates and scheduled retros to discuss what went wrong and what went well.

Over time we discovered that through the use of our own tool, we tackled a notoriously difficult problem: changing developer behavior in ways that make the team more productive while easing the administrative tasks that usually slow them down.

Dogfooding our own extension has helped our team develop a better shared understanding of the code base, and integrate the non-coding tasks required for healthy project management into the dev-loop so tightly that it becomes the preferred way for developers to work.

We’ve realized that if you build tooling that’s fun to use, this will all happen organically.

Iterating in the Dark

The thing about silos is that they’re usually dark inside.

When we first started building our VS Code extension for Bitbucket and Jira users, we were working in a very familiar style where we each went into our “coding caves” for long periods of time and every once and a while came up for air to check the Slack channel we had setup.

It was almost guaranteed that there was a message in the channel begging for someone to review a PR from an hour or two ago, if not more. Then, as any good team member, we’d stop what we were doing, open up Bitbucket and review the code in our browsers. We might add a comment here and there (to be forgotten by it’s author as soon as it was submitted), then certainly take a coffee break, and then come back to crawl into our caves again.

Rinse, Repeat. The problem was, it still felt like separate people working on separate areas of the code base without a lot of cross-functional knowledge of how things worked. We were basically in the dark.

To shed some light (pun intended) on some of the issues:

Nobody likes sitting in a room for hours with a Jira board on the screen while somebody frantically types trying to enter Jira issues while heated discussions conversations are happening.
Once coding commences, developers tend to only take the tasks within their comfortable domain knowledge (silos) and cross-functional learning is lost
The lack of cross-functional code base knowledge means bugs and ideas about other parts of the code base are left up to the developer who works on it lacking collaborative input
Developers hate context switching which in-turn means:
bugs or features may be discovered, but new Jira issues don’t get entered for them
pull requests are submitted, but nagging has to happen to get them reviewed
comments may be entered on a pull request, but replies are either not followed up on or take days to get new responses
updating Jira issues to reflect the state of the project is a chore that happens through nagging, thus leaving Jira in a state of lies
Ensuring Jira units of work are linked to code in every branch and every commit is not consistent at best and doesn’t happen in the worst cases
There’s less collaboration when developers feel like someone else is the expert

Out of the Caves

One day, it happened. We had finally hit the point where we could authenticate against both Jira and Bitbucket and we had the essential features in place to start dogfooding our extension. That first developer exclaimed with triumph “Hey, I just approved that PR with our extension!” and we knew feasibility was no longer our mission…

The new roadmap, almost entirely led by engineers with little oversight and lots of dogfooding, emerged: we would take a selfish approach and focus on making our own lives easier inside of VS Code, knowing that if we got the Jira and Bitbucket experience (mostly) right, we would have a positive impact on other development teams outside of Atlassian. If we have pain points in our development cycle, chances are other teams do as well.

To illustrate some of the successes our team has enjoyed, let’s take a look at our new development practices which integrate the features of our extension to blur the lines between coding, Jira-ing and Bitbucket-ing. And what better place to start than… the middle.

A Shared Cave with Really Nice Lighting

Jira Units of Work: Never Lose Track of Discoveries

OK, let’s say I’m a developer… and I’m working on some code when I notice a bunch of comments littered with Jira issue keys. (we’ll get to why that is in a bit) It’s a bit frustrating to see the keys and a small comment but not have access to the issue details. It’s even more frustrating to hop out of my IDE to paste the key into the Jira web interface just to see the details, and so I add a little TODO comment:

// TODO: add a "quick view" when hovering over Jira issue keys

Even though there are various extensions to list/manage TODO comments, this is going to get lost if I don’t remember to bring it up at the next planning meeting, or stop what I’m doing, jump into the Jira web interface and create an issue for it.

Using the “Create Jira Issue” code link that appears for customizable comment prefixes (TODO, BUG, FIME, ISSUE, etc), I can simply click on the link, create the issue and move along with my coding.

To top it all off, the extension will update the comment and add the newly created issue key as a prefix so other developers can reference it.

// TODO: VSCODE-12324 - add a "quick view" when hovering over Jira issue keys

Turning Ideas into Code: Starting Work

Let’s say there’s another developer on the team that just finished a task and is looking for something to pick up. She sees the Jira issue we made above in her Jira Issue Tree within VS Code and opens it up. After reading through the details she decides she wants to work on it.

Following our best coding practices, getting started on a new task is a little more complicated than just coding away. She needs to:

Create a branch ensuring that the Jira issue key is in the branch name
Link the local branch with a new upstream branch on Bitbucket
Assign the issue to herself
Transition the issue to an “In Progress” state so other developers and “project people” know what’s in flight
Finally checkout the local branch and start coding

This is a small list of things that needs to be done, but after doing it a few (hundred) times, most developers tend to miss a few of these steps.

Using the “Start Work On Issue” button provided on the Jira Details screen within VS Code, this can all happen in a single step.

Pro Tip: Make sure all commits contain the issue key

So what’s this about putting issue keys in branch names?

The idea is simple: if you link your Jira instance in your Bitbucket repository settings, Bitbucket will look for issue keys in your branch names and commit/PR comments and be able to link them in the Bitbucket UI, among other things.

On top of the Bitbucket UI, the Atlassian for VS Code extension checks those same places for issue keys and “magically” gives you lists of issues related to your PRs within various UIs. You can now easily get a better picture of issue/code relationships right within VS Code.

So how to you ensure every commit has a Jira issue key in the comment?

Although outside of the VS Code extension, we use a small Git script that automates the process by finding the issue key in your branch name and pre-pending it to all commit comments on that branch.

Developers never have to type in the issue key in a comment as long as the branch contains the key.

You can grab this nifty script and instructions from our Bitbucket Snippet.

So our developer feels like her feature is complete and now needs to create a pull request. This is going to serve multiple purposes:

Her teammates will do a code review to catch anything she might have missed
Her teammates will have the opportunity to test out the new feature
The entire team can discuss the approach and make any changes as needed
The team has a chance to “sign off” on the pull request by approving it
Finally, the pull request can be merged

Let’s take this step-by-step, first by creating a pull request.

Creating a pull request traditionally meant hopping out of your IDE, navigating the Bitbucket UI, and creating the pull request. With our extension, this can now be done right within VS Code.

A pull request has been created and now is typically when our developer takes (a much deserved) coffee break and when she gets back, checks to see if anyone’s looked at it. Usually the answer is no and so she needs to hop into Slack and gently request that her teammates take a look at the PR. Even this requires some luck that the other developers are in one of their “come up for air” moments.

In our extension we wanted to try to remove as much nagging and waiting as possible, so when she created the pull request, her teammates got a small notification popup right within VS Code and her pull request showed up in all of their “Pull Request Tree” views right within VS Code.

Now her teammates not only know that there’s something to review, but they can see the detailed summary of the pull request, see any and all Jira issues that a related to the pull request, and go through the individual file diffs right withing their IDEs.. you know, where developers like to look at code in whatever crazy theme they have decided to use that week.

As they go through the details and the diffs, they can quickly add comments on a line-by-line basis, and since developers are notified as new comments are added within VS Code, the time spent waiting for replies is greatly reduced, effectively turning pull request into a meaningful communication tool.

Since using these pull request features, our team has enjoyed a huge acceleration in knowledge sharing which in-turn has help break down our silos and turn each team member into a product expert instead of an expert of a smaller domain within the code base.

We all have a deeper understanding of our entire code base and have also become a lot more collaborative simply through greater communication.

Once each teammate finishes reviewing the pull request, they can simply click the approve button from the details screen within VS Code and go back to their coding. Similarly, our developer that submitted the PR can simply merge right from the details screen as well.

Once merged, it’s time for a new task. Our developer can either go through the list of issues in the tree view like before, or now she can simply hover over an issue key she’s found in the code and use the new “Issue Quick View” feature which will allow her to get essential issue details, open up the full detail view and optionally start work on the issue.

Versioning is a Result, Not a Plan

Our team has come a long way since deciding to test out using Bitbucket and Jira features inside of VS Code. We set out to create a useful tool to help ease some of the pain points within common coding cycles. What we ended up with was exactly that. What we didn’t plan on, was that through the use of our own tool, we would organically change the way we worked.

The biggest change we’ve seen so far is that we no longer go into a room armed with a version number for the “next release”, give it a date, and pack it with things we think we can get done. Instead, we code, we discover bugs and features along the way, we are more collaborative in our development, and everyone now feels ownership in the entire code base and are eager to work on any part of it.

Where we used to do a release at some scheduled time period, we now do many more releases as useful features are completed. We don’t wait to ship anything because “1.x is supposed to release at the end of the month”. In fact, we only have a single version we ever work towards and it’s label is vNext. At any point in time, we can say “it’s time.” and then minutes later we release and assign that snapshot in time an appropriate label.

When there’s no deadline, you’re never done

Our team is moving faster than ever and we’ve already see vast improvements to our coding cycles.

By no means is our extension a “silver bullet” and as they say “results may vary”. Our team is passionate (and selfish) about continuously finding ways to use new tooling and new workflows to make every team the best they can be.

We’re excited to learn how teams outside of Atlassian make use of these features and how we can improve.

Install the extension today and let us know what you think!

Written by Jonathan Doklovic: Jonathan has been working at Atlassian for almost a decade and is currently a Principal Developer on the Product Integrations team. He has worked on many Atlassian products as well as the core plugin system and Atlassian Connect.

Originally published at https://bitbucket.org on May 15, 2019.

LAMP vs. MEAN: Which stack is right for you?

Ash Moosa — Thu, 02 May 2019 16:57:51 +0000

A web stack is a collection of software or technologies that are used to build a web application. Choices are plenty, but picking one can be hard.

When chatting with co-workers, developers or customers, suggestions for what technologies and stacks to use couldn’t be more different. When I started off as a web developer, I went the usual way at that time: learning HTML & CSS, exploring some PHP — and of course MySQL. That was, if you were not using Java or ASP.NET, the technology stack of that time. Whether you wanted to host a blog, a bulletin board or become an image hoster — you would more than often need these things: Linux, Apache, MySQL and PHP (LAMP).

Here is a detailed overview of LAMP and the relatively new, MEAN stack, which are currently the most popular open source web stacks and a brief overview of other stacks. Whichever stack you choose, Bitbucket works with them all.

LAMP

LAMP delivers a strong platform for developing and hosting large, performant web applications. With the biggest and oldest community, countless libraries and tools, you get great support and will find developers quite easily.

Its integral components are:

Linux (OS)

Apache (Webserver)

MySQL (Data persistence)

PHP (Programming language)

There are also some derivatives of this stack:

LAMP (with Perl or Python instead of PHP)
LAMP (with MongoDB instead of MySQL)
WAMP (Windows as OS)
MAMP (Mac OS X as OS)
XAMPP (Any OS + Perl or PHP + FTP Server)
LAPP (PostgreSQL as database)

Pros:

LAMP is kind of the dinosaur of web development, used by hundreds of thousands of companies and therefore maintained and supported very well. With endless modules, libraries and add-ons available you can adapt it to your company’s needs.

Being Linux based, you will find help for any topic in the large open source community. MySQL is a very reliable and scalable solution. PHP is in version 7 and is also supported by a mature and big community. PHP is also very fast and integrates well with the rest of the stack.

You can control the server and decide which versions and software you install, so you don’t have to depend on the client’s browser. Best for if you have lots of server-side tasks.

Cons:

Because it’s easy to learn, there are a lot of developers out there who are not following best practices and building garbage apps. Starting with PHP is easy, but mastering it is hard. This is also true for security in these PHP apps. Some would also describe it as a script language instead of a real programming language because it’s not strongly typed and not pre-compiled. I’d recommend diving in deeper into pros and cons of PHP, Python or Perl.

As for MySQL, other options are becoming more mature. NoSQL databases like MongoDB are popular among enterprises today due to it’s scalability. Plus, pure JavaScript Stacks like MEAN gain more traction every year and new developers might not be interested in learning all of the LAMP’s skills.

MEAN

Compared to LAMP, the MEAN stack is fairly new. One of its biggest differences is that MEAN is not dependent on a specific operating system — Node.js takes care of server-side execution. The MEAN Stack is especially recommended for JavaScript enthusiasts — as it uses JavaScript at all levels. This also makes it preferred by new developers.

MongoDB is a popular and flexible document based, NoSQL database, compared to MySQL’s relational database system. Angular helps build progressive and modern web apps.

Its components are:

MongoDB (Data persistence)

Express.js (server-side application framework)

Angular.js (client-side application framework)

Node.js (server-side environment)

This stack has some derivatives too:

MERN (React instead of Angular)
MEEN (Ember.js instead of Angular)

Pros:

Using JavaScript as the primary programming language is a huge advantage. Everything can be set up quickly and done in JS, which makes it much easier to find developers, and LAMP developers typically know JavaScript as well. MongoDB is very popular for its easy schemaless data persistence and is faster than MySQL if you have a lot of read requests. The fact that Angular is maintained by Google is also a big plus. It receives new releases and functions on a constant basis. Another huge advantage is the ability to easily build mobile or desktop apps, for example with Ionic. Code and components can easily be reused or added.

Cons:

Like all new technologies, MEAN’s glamour is creating some hype. Developers fall for this hype and build their apps in JavaScript, just because it’s trendy. Many of these libraries and frameworks are quite new, and new versions get released quickly, so maintaining your app can become quite a hassle. Since many technologies disappear after a few years, sustainability can become an issue. It’s also harder to maintain a clean code base and follow best practices as your app grows. Further, you have to rely on the client and the client’s available technologies e.g. if you are targeting IE users, embedded systems or low end PCs, there may be usability issues.

A few other stacks to consider:

WISA Windows Server / IIS / Microsoft SQL Server /

Not open source, but all components are from Microsoft, so it should work seamlessly.

LAMP (With MongoDB) Linux, Apache, MongoDB, PHP

NoSQL Databases like MongoDB can also be used in a classic LAMP environment.

Ruby Stack Ruby/Ruby on Rails/RVM (Ruby Virtual Machine) / SQLite

This stack is losing popularity. Ruby on Rails was an often used framework once, and thus the whole stack.

Java+Spring

Preferred by large enterprises and shied by indie developers for its complexity, Spring offers an entire full-stack framework written in Java.

Django Stack Python / Django / Apache / MySQL

The Django framework is loved by Python developers, delivers performance and is often referred to as an easy to learn stack.

Which stack is used more frequently?

It’s hard to compare the popularity of stacks, but you can useGoogle Trends to compare programming languages and get a feel for what people are searching for. As the chart below shows, JavaScript is searched for more than PHP right now.

I’d recommend checking development trends using Google’s trend tool from time to time.

I’d also suggest diving deeper on databases (SQL vs. NoSQL) to gain a basic understanding of the two concepts and make a choice.

So, how do you pick a stack?

Picking a stack depends on many factors. If you are a developer or project owner, here are a few questions to ask yourself.

What kind of web-application am I planning to create?
What is its expected lifetime?
What technologies are available at my customer’s/client’s /cat’s/… infrastructure?
How easy is it to find developers to maintain the application?

Let me give you an example. Let’s say you have a website for listing used cars. It was developed using a LAMP stack a while ago. But your website lacks a back-end for used car dealers, where they can manage their listings on your website. Depending on your company size, time and budget, you have to ask yourself all the above questions. If you have a small team, it might make sense to extend your existing application in the LAMP environment. Since your developers know the ecosystem and it’ll be much faster. If you have time and resources to spare, you could take another approach and extend your existing LAMP application with an API. Later, your team could focus on developing a small, standalone (M)EAN application, that can easily be maintained, improved with new features and released using a much faster cycle.

Another example: You want to build a newsletter platform, where people can sign up, upload mailing lists, compose mailings and so on. You could of course use MEAN, but you have large scale and high traffic potential. It may make more sense to use a LAMP stack as your foundation, since Linux, MySQL and Apache provide a stable, scalable environment with lots of community support for any thinkable problem. You will also have lots of server-side tasks and cronjobs and will encounter mailing topics like SMTP and so on. I would recommend a Linux environment customized to your needs in this case.

Here is a summary of things to know/consider.

MEAN

Single code base (JavaScript)
Popular for modern web apps and hybrid apps
Supported by large companies like Google
Better for apps where a lot of the logic happens on the client’s side
Harder to maintain long term due to the rapidly evolving javascript ecosystem
Best for progressive web apps

LAMP

Better for large applications
More mature, huge community
Well-established application frameworks like Symfony, Zend, Laravel
Easier to follow standards and easier to keep code clean

If you are new to programming and web development, ask yourself:

What is the easiest to learn for you and your team?
What technologies are trending and which will win in the long run?
If open source, could you imagine contributing to the project?
Which technologies will serve you personally in the long term?

A great resource for finding answers on JavaScript technologies is the StateOfJS project — https://stateofjs.com/ — it’s a project that conducts a survey every year asking thousands of developers on their opinions on current technologies and salary.

This is a guest post for Bitbucket by Christoph Heike — “I’ve been developing web applications for over 10 years and currently run a web-development agency in Bonn, Germany. I’m also involved in an Amazon-based tech startup. My goal is to always deliver clean, sustainable and high performant software solutions. Connect with me on LinkedIn.”

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on May 2, 2019.

Deploying an Angular app on a Google VM Using Bitbucket Pipelines

Ash Moosa — Tue, 30 Apr 2019 17:27:56 +0000

This is a guest post written for Bitbucket by Suren Konathala.

Angular is one of the most widely used javascript frameworks. But though the builds are easy, developers face issues when configuring deployments and setting up CI/CD pipelines. This post outlines the steps required to deploy an Angular application to a Google VM using Bitbucket Pipelines.

What are Pipelines?

Bitbucket Pipelines allows developers to configure continuous delivery (in the cloud) of source files to test/production servers. These pipelines are configured to connect to the production server using YML scripts.

Why I used Pipelines?

For users to be able to access the application, the source code needs to be deployed to a server. The server from which the web application is rendered/delivered to users is called a Production server. And before the application reaches the production server, it goes through many iterations of development and testing. These iterations are usually deployed to a Development server or a Stage server.

For an application to be deployed to each of the above servers, there are several steps involved that can get cumbersome.

Copy the code files to the server
Run the build and deploy scripts
Repeat the same on each server. Sometimes teams have multiple servers for each stage.

Okay! The above has been automated to some extent by tools like Jenkins but the drawback is that developers/admins have to install another software on top of their servers, and learn to use and administer it.

Pipelines simplifies the above process, and in fact automates the entire build and deploy process — also known as CI/CD (Continuous Integration / Continuous Deployment). The best part with Bitbucket Pipelines is that applications can be built and deployed directly from the Bitbucket repository to any destination server.

The entire copy, build and deploy process can be defined using simple YAML based scripts without the need for any additional software or a solution. All we need to do is to pick the docker image (like NodeJs, Java, etc.) for the pipeline to use to build your project and select the frequency (e.g. manual, or automatically when files are updated in the source repository). This saves a lot of time and resources for teams and organizations.

Tutorial: Configure an Angular Project on Bitbucket

This tutorial covers how to deploy an Angular based web application to a Google Cloud Virtual Machine (VM). Source code for the application is in a Bitbucket repository and this VM will need to be connected to using SSH security keys.

Pre-requisites

External server (or VM) with private and public SSH keys. This will the hosting machine for the website or web application.
Repository onBitbucket with project sources files. These will be used to build & deploy to the server.

Step 1: Setup SSH Keys

Under Bitbucket > Project source repository > Settings > Pipelines > SSH Keys

Add private and public keys. You need to get these from the external server that you need to connect to.

Add Known hosts (this will be the IP address of the external server you want to push the code to. In our example, we used a VM on Google Cloud)

Step 2: Define the YML deployment script

Go to Project source repository > Pipelines > New pipeline and define the script. Here is an example script:

# Sample build file # @author Suren Konathala 
# ----- 
image: node:8 
pipelines:   
  default: 
- step:      
     caches:        
     - node      
     script: # Modify the commands below to build your repository.
       - echo "$(ls -la)"        
       - npm install        
       - npm install -g @angular/cli        
       - ng build --prod        
       - echo "$(ls -la dist/)"        
       - scp -r dist/ user@34.73.227.137:/projects/commerce1

The above script performs the required commands/steps to build an Angular project. Once done, it pushes/deploys the contents of the build files under the dist folder to an external server.

In this example, we used an SCP command to push code to an external server. Since the SSH keys are already set in step 1 above, your local repo can now connect to the server and copy the files over.

Step 3: Run the pipeline

Save and “run the pipeline”. You can see the running log and status of the pipeline.

On successful completion, you can verify if the files actually being copied to the external server by either by testing your live application and checking if the latest updates are reflected or you can manually check if the files on the server were updated (no need to do this for every push).

Troubleshooting/Info

To update NodeJS version in the pipeline.. change the version of the docker node image to node:8 in the YML script.

image: node:8

To know what folder you are in and what files are being generated, you can use echo commands. Some examples..

echo "Starting the deployment..."
echo "$(ls -la)"

This post shows how to add SSH Keys on a Linux server/VM.

The source files for this project are shared here.

Learn more about building Angular projects here.

This is a guest post written for Bitbucket by Suren Konathala. “My mission is to simplify technology adoption for organizations. I’m a developer, architect, consultant and manager. I love to write and talk about technology. Connect with me on LinkedIn or Twitter.”

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on April 30, 2019.

Bitbucket + Bitrise: Configuring Continuous Integration for an iOS app

Ash Moosa — Mon, 29 Apr 2019 17:03:05 +0000

This is a guest post written for Bitbucket by Ivan Parfenchuk. Ivan is an independent iOS and Ruby developer passionate about building delightful experiences. Connect with him on Twitter @gazebushka.

When iOS applications start growing, at some point it becomes essential to have a quick develop-release-test feedback loop. You can create this loop by doing everything manually, but it can be much quicker and more advanced if you use Continuous Integration (CI) tools.

With a CI tool, you can build up a history of releases and quickly see which build contained what. You can run tests for every build automatically and catch some inevitable bugs. You can have consistency in your release notes. And you can streamline your release cycles, which automates your checklists.

Sound interesting? Let’s try to build this feedback loop using Bitbucket Webhooks, Bitrise and fastlane.

Deployment flow

The flow we are going to use for our Continuous Integration is going to look like this:

Create & merge a Pull Request in Bitbucket
Bitbucket performs a “Webhook” HTTP request to Bitrise
Bitrise starts building the process and launches fastlane
fastlane builds the app and sends it to App Store Connect
App Store Connect processes the build, and it becomes available in TestFlight

Bitbucket Webhooks and git branching model

Each deployment starts from us creating a Pull Request.

Let’s say your team is using master git branch for code in the releasable state. It also makes new releases by merging this master branch to the release branch.

The following section describes how to create a Webhook manually. However, if you use Bitrise, it can create a Webhook for you automatically, so, feel free to skip to the Bitrise section.

Manual Webhook configuration

Next, let’s configure Bitbucket Webhooks so that whenever someone pushes to release branch or merges Pull Request to release branch, the Webhook is triggered.

To do that go to your Bitbucket repository and click “Settings” in the side menu. Then click “Webhooks” in the “Workflows” section and then click “Add webhook.”

Fill out Title, URL (see below), set Status to Active, and select “Choose from a full list of triggers” for Triggers. The triggers we are going to use are:

Repository: Push
Pull Request: Created, Updated

To get the URL for our Webhook:

head over to Bitrise, create a new app
open the Dashboard -> Your app -> Code tab
scroll to Incoming Webhooks section and click Setup Manually.
Select “Bitbucket Webhooks” and copy the Webhook URL

Bitrise and automatic Webhook configuration

Bitrise is a platform for Continuous Integration. You can configure different deployment “workflows” in it and have the Bitrise servers build and publish your application. Here are the steps to create a new deployment workflow for our CI setup.

First sign up in Bitrise, go to Dashboard, click “Add New App”
Select “Private” if you want your config and logs to stay private
Select Bitbucket and connect it to your account
Click “Auto-add SSH key” or configure SSH access manually
Enter release as branch name in the “Choose branch” step
In Project build configuration select “fastlane” , check that Fastlane lane is set to ios release.
Select the stack that you normally use to build your app or just the latest available Xcode/macOS and click Confirm.
In the last step “Webhook setup” click “Register Webhook for me”

The last step creates a Webhook in Bitbucket, so you don’t have to do anything manually. You can head over to Bitbucket repository and check the Webhook configuration in Settings -> Webhooks.

In our setup, we are going to use fastlane to build and publish the app to App Store Connect.

Fastlane configuration

Fastlane is a set of tools for automating development and release process.

Follow this guide to install fastlane:Setup — fastlane docs

In short, you need to install Xcode development tools:

xcode-select --install

and then install fastlane via RubyGems

sudo gem install fastlane -NV

or via brew:

brew cask install fastlane

Then open the working directory of your app in the Terminal and initialize fastlane.

cd /path/to/your/app 
fastlane init

Select “3. 🚀 Automate App Store distribution”

Then follow the configuration requests. Fastlane can create and configure the new App Id for you and create a sample deployment “lane.” Lane is just a collection of steps, required to complete some scenario.

Once the configuration is finished, let’s open the Fastfile which has been created and configure our first deployment script. Fastlane has large set of tools for automating various processes like code signing, uploading of screenshots, running tests and so on. However, we are going to start with a simple setup:

default\_platform(:ios)  

platform :ios do   
  desc "Push a new release build to the App Store"   
  lane :release do     
    build\_app(scheme: "CITest")     
    upload\_to\_app\_store(force: true, skip\_metadata: true, skip\_screenshots: true)   
  end 
end

The “release” lane will

perform a build of your project build_app(scheme: “CITest”)
Upload the resulting ipa file to App Store Connect upload_to_app_store. In this guide, we are skipping Fastlane metadata upload.

You can test your setup by opening your project directory in Terminal and running fastlane release :

cd /path/to/your/app/directory 
fastlane release

Code-signing

If you see problems with code-signing start here:Troubleshooting — fastlane docs. You can use fastlane match to manage code signing, but be careful: if you already have generated Certificates and Provisioning Profiles, match can break things. However, if it’s a completely new setup or you don’t care much about the existing profiles, match is going to speed things up considerably.

We’ll use fastlane match in our example:

cd /path/to/your/app/directory 
fastlane match development 
fastlane match adhoc 
fastlane match appstore

Then open the Xcode, turn off Automatic code signing and select provisioning profiles that match has generated.

After that, we can add match to our Fastfile:

default\_platform(:ios)

platform :ios do 
  desc "Push a new release build to the App Store" 
  lane :release do 
      match(type: "appstore", readonly: true) 
    build\_app(scheme: "CITest") 
    upload\_to\_app\_store(force: true, skip\_metadata: true, skip\_screenshots: true) 
  end 
end

Finishing up the Bitrise setup

Bitrise is building the project on its servers which don’t have any of your passwords and credentials required to code-sign and upload your app to App Store Connect. Therefore we’ll have to share some of those, precisely these two:

Login/password for App Store Connect user
Password to decrypt your match repository (if you use match)

The App Store Connect user doesn’t have to be the one you use to control your apps. You can create a new user in App Store Connect, which only has access to the app you automate and has at least a Developer role.

Once you set your new App Store Connect user, head over to Bitrise and open Workflow Editor tab and then Secrets. Add two new secrets:

ITUNES_CONNECT_USER and ITUNES_CONNECT_PASSWORD with App Store credentials for this new user. Plus put the same password into FASTLANE_PASSWORD secret.

If you use match, then add one more secret called MATCH_PASSWORD with the password you used to encrypt match repository.

Conclusion

That should be it. Try to create a new Pull Request and merge it and see if Bitrise triggers the new build. If everything goes well, you will see the new build in TestFlight and will be able to select it for your new iOS app version.

There is much more that you can do with automated deployments, such as:

Testing using a fastlane scan
Automated build number incrementation
dSYM uploads to Crashlytics_Raygun_etc

However, start with simple things first. I hope this guide helps you!

Love sharing your technical expertise? Learn more about the Bitbucket writing program.

Originally published at https://bitbucket.org on April 29, 2019.

Searching DynamoDB: An indexer sidecar for Elasticsearch

Ash Moosa — Thu, 28 Mar 2019 22:43:02 +0000

TLDR;

DynamoDB is great, but partitioning and searching are hard
We built alternator and migration-service to make life easier
We open sourced a sidecar to index DynamoDB tables in Elasticsearch that you should totes use. Here’s the code.

When we embarked on Bitbucket Pipelines more than three years ago we had little experience using NoSQL databases. But as a small team looking to produce quality at speed, we decided on DynamoDB as a managed service with great availability and scalability characteristics. Three years on, we’ve learnt a lot about how to use and how to not use DynamoDB, and we’ve built some things along the way that might be useful to other teams or that could be absorbed by the ever growing platform.

To NoSQL or not to NoSQL that is the question

First off the bat, relational databases aren’t werewolves and NoSQL isn’t a silver bullet. Relational databases have served large scale applications for years and they continue to scale well beyond many people’s expectations. Many teams in Atlassian continue to choose Postgres over DynamoDB for example and there are plenty of perfectly valid reasons to do so. Hopefully this blog will highlight some of the reasons to choose one technology over the other. At a high level they include considerations such as operational overhead, the expected size of your tables, data access patterns, data consistency and querying requirements.

Partitions, partitions, partitions

A good understanding of how partitioning works is probably the single most important thing in being successful with DynamoDB and is necessary to avoid the dreaded hot partition problem. Getting this wrong could mean restructuring data, redesigning APIs, full table migrations or worse at some time in the future when the system has hit a critical threshold. Of course there is zero visibility in to a table’s partitions — you can calculate them given a table’s throughput and size but it’s inaccurate, cumbersome and we’ve found, largely unnecessary if you’ve designed well distributed keys as the best practices developer guide suggests. Fortunately we took the time to understand partitioning from the get go and have managed to avoid any of these issues. As an added bonus, we’re now able to utilize autoscaling without concern for partition boundaries because requests remain evenly distributed even as partitions change and throughput is redistributed between them.

Throughput, bursting and throttling

Reads and writes on DynamoDB tables are limited by the amount of throughput capacity configured for the table. Throughput also determines how the table is partitioned and it affects costs so it’s worth ensuring you’re not over provisioning. DynamoDB allows bursting above the throughput limit for a short period of time before it starts throttling requests and while throttled requests can result in a failed operation in your application, we’ve found that it very rarely does so due to the default retry configuration in the AWS SDK for DynamoDB. This is particularly reassuring because autoscaling in DynamoDB is delayed by design and allows throughput to exceed capacity for long enough that throttling can occur.

DynamoDB auto scaling modifies provisioned throughput settings only when the actual workload stays elevated (or depressed) for a sustained period of several minutes. The Application Auto Scaling target tracking algorithm seeks to keep the target utilization at or near your chosen value over the long term.

Sudden, short-duration spikes of activity are accommodated by the table’s built-in burst capacity.

You’ll still want to set alerts for when throughput is exceeded so you can monitor and act accordingly when necessary (e.g. there’s an upper limit to autoscaling) but we’ve found that burst capacity, default SDK retries, autoscaling and adaptive throughput combine extremely effectively such that intervention is seldom required.

Source

Alternator: an object-item mapping library for DynamoDB

After Pipelines got the green light and refactoring of the horrendous alpha code began, one of the first things we built was alternator: an internal object-item mapping library for DynamoDB (similar to an ORM). Alternator abstracts the AWS SDK from the application and provides annotation based, reactive (RxJava — although currently still using the blocking AWS API under the covers until v2 of the SDK becomes stable) interfaces for interacting with DynamoDB. It also adds circuit breaking via Hystrix removing much of the boilerplate code that was present in early versions of the system.

@Table(
    name = “pipeline”,
    primaryKey = @PrimaryKey(hash = @HashKey(name = “uuid”))
)

@ItemConverter(PipelineItemConverter.class)
public interface PipelineDao {
    @PutOperation(conditionExpression = “attribute\_not\_exists(#uuid)”)
    Single<Pipeline> create(@Item Pipeline pipeline);
}

The migration service

DynamoDB tables are of course schema-less, however that doesn’t mean that you won’t need to perform migrations. Aside from a typical data migration to add or change attributes in a table, there are a number of features that can only be configured when a table is first created such as local secondary indexes which are useful for querying and sorting on additional attributes other than the primary key.

The first few migrations in Pipelines involved writing bespoke code to move large quantities of data to new tables and synchronizing that with often complex changes in the application to support both old and new tables to avoid downtime. We learnt early on that having a migration strategy would remove a lot of that friction and so the migration service was born.

The migration service is an internal service we developed for migrating data in DynamoDB tables. It supports 2 types of migrations:

Same table migrations for adding to or amending data in an existing table.
Table to table migrations for moving data to a new table.

Migrations work by scanning all the data in the source table, passing it through a transformer (that is specific to the migration taking place) and writing it to the destination table. It does this using a parallel scan to distribute load evenly amongst a table’s partitions and maximize throughout to complete the migration in as short a time as possible.

Table to table migrations then attach to the source table’s stream to keep the destination table in sync until you decide to switch over to using it. This allows the application to switch directly to using the new table without having to support both old and new tables during the migration.

Querying in DynamoDB: a tale of heartache and misery

DynamoDB provides limited querying and sorting capability via local and global secondary indexes. Tables are restricted to five local secondary indexes, with each index sharing the same partition key as the table’s primary key, and a query operation can only be performed against one index at a time. This means that on a “user” table partitioned by email address, a query operation can only be performed in the context of the email address and one other value.

Global secondary indexes remove the partition key requirement at the cost of paying for a second lot of throughput, and only support eventually consistent reads. Both types of indexes are useful and sufficient for many use cases and Pipelines continues to use them extensively but they do not satisfy the more complex querying requirements of some applications. In Pipelines this need predominantly came from our REST API which rather typically allows clients to filter and sort on multiple properties at the same time.

Source

Our solution

Our first attempt at solving this problem, MultiQuery, was built in to an alternator. With this approach we queried multiple local secondary indexes (LSI) and aggregated the results in memory allowing us to perform filtering and sorting on up to five values (the maximum number of LSI) in a single API request. While this worked at the time, it started to suffer from pretty terrible performance degradation as our tables grew in size and exponentially so the more values you were filtering on. Before multi query was replaced, the pipelines list page on our end-to-end test repository (one with a large number of pipelines) took up to tens of seconds to respond and consistently timed out when filtering on branches.

A common pattern for searching DynamoDB content is to index it in a search engine. Conveniently AWS provides a logstash plugin for indexing Dynamo tables in Elasticsearch so we set about creating an indexing service using this plugin and the results were encouraging. Query performance vastly improved as expected but the logstash plugin left a lot to be desired taking almost 11 hours to index 700,000 documents.

Some analysis of the logstash plugin and the realization that we had already built what was essentially a high performance indexer in the migration service led us to replace the logstash plugin with a custom indexer implementation. Our indexer, largely based on the same scan/stream semantics of the migration service and utilizing Elasticsearch’s bulk indexing API, managed to blow through almost 7 million documents in 27 minutes.

The indexer sidecar

The custom indexer has since been repackaged as a sidecar, allowing any service application to seamlessly index a DynamoDB table in Elasticsearch. Both the initial scan and ongoing streaming phases are made highly available and resumable by a lease / checkpointing mechanism (custom built for the scan, standard kinesis client for the stream).

We currently utilize the excellent elasticsearch client built by the Bitbucket code search team to query the index and have started work on an internal library which adds RxJava and Hystrix in the same vein as alternator.

Here is the repo with the code and a readme.

Final thoughts

If you haven’t used NoSQL before, it certainly requires a mind shift but overall our experience with DynamoDB has been a positive one. The platform has proven to be extremely reliable over the past three years (I can’t remember a single major incident caused by it) with our biggest challenge coming from our querying requirements. Some might say that’s reason enough to have chosen a relational database in the first place and I wouldn’t strongly disagree with them. But we’ve managed to overcome that issue with a solution that is for the most part abstracted away from day to day operations. On the plus side, we haven’t had to run an explain query in all that time or deal with poorly formed SQL, complex table joins or missing indexes and we’re not in a hurry to go back.

This post was written by Sam Tannous. Sam is a senior software engineer at Atlassian and is part of the team that brought CI/CD Pipelines to Bitbucket cloud. With over 15 years industry experience he has co-authored 3 patents, maintains 1 open source project and has a track record of successfully delivering large scale cloud based applications to end users. Connect with him on linkedin.

Originally published at bitbucket.org on March 28, 2019. To contribute to the Bitbucket blog, apply here.

Beer, Bravado & Bitbucket: Using data to drive CODE decisions

Ash Moosa — Tue, 26 Feb 2019 21:47:28 +0000

Product teams and marketing teams continually use data to drive decisions. But what about us as software engineers? What can we do with metrics, besides just pulling them in for somebody else?

“You mentioned beer. I was led to believe there would be beer.”

Source: Guinness.com

People like Guinness… a lot. By 1914, their annual output was almost a billion pints.

That’s going to need a lot of quality control. So clever Mr. Claude Guinness decided to hire smart graduates to do this work, sort of like tech companies do now.

See the guy in the photo? That’s Billy. Or William Sealy Gosset to his less-intimate friends. “An energetic — if slightly loony — 23 year-old scientist”.

Source: Wikipedia

To help him maintain quality over a billion pints, our friend Billy invented a way to answer important questions, by taking small samples of beer and using some smart math. The methods invented by our clever Billy became the fundamental basis for how doctors decide if a new drug will save you(or kill you), how Facebook decides how to manipulate you, and a zillion other out of this world uses.

“What does this have to do with code?”

Bitbucket (and the rest of the company) has a big push on performance. On the frontend, we’re making a number of changes to boost performance of the Pull Request page. In particular we want to speed up rendering the code diffs.

Rendering the diffs uses a Chunks component which renders a great many separate CodeLinecomponents.

I wondered, could we speed this up by merging the CodeLinemarkup directly into Chunks?

But that reduces modularity and possibly maintainability. Since there’s a cost, I want to know there’s a real benefit to rendering time. But how do I tell? All I have so far is an idea.

Sure, I could load the Pull Request page a couple times before and after the change. But that’s hardly a rigorous check, for many, many reasons. And it’s going to be hard to tell, if the improvement is small (but real).

“Now tell me more.”

I needed something better than refreshing my browser a couple times. So, I used Lighthouse, an awesome tool for measuring frontend performance and collecting metrics. And it’s now available for the command line.

I wrote a ‘custom audit’ that let me measure diff rendering times for the Pull Request page, and a batch tool for executing multiple runs. Hooray for reliable frontend metric gathering!

In this instance, the audit measures our diff rendering time. This is shared for everyone’s enjoyment (everyone loves code right?) but bear in mind this was written as an internal dev tool. Production code would have things like nice error handling. Also note that this is for v3.2.1 of Lighthouse.

First, the code you are measuring should mark and measure User Timing events for Lighthouse to pickup:

performance.mark("DIFFSET.RENDER.START");

(...time passes..)

performance.mark("DIFFSET.RENDER.END");
performance.measure("DIFFSET.RENDER.DURATION", "DIFFSET.RENDER.START", "DIFFSET.RENDER.END");

Next, the custom audit.

// diff-rendering-only-audit.js

'use strict';

const { Audit } = require('lighthouse');

// Custom audits are provided to Lighthouse as classes
class DiffRenderingOnlyAudit extends Audit {

 // Tell Lighthouse about this audit
 static get meta() {
 return {
 id: 'diff-rendering-only-audit',
 title: 'Diffs rendered',
 failureTitle: 'Diff rendering slow',
 description: 'Time to render a diffset',
 requiredArtifacts: ['traces'],
 };
 }

 // Lighthouse will call this static method
 static audit(artifacts) {
 return new Promise(resolve =\> {
 getAuditResult(artifacts, resolve);
 });
 }
}

// Lighthouse produces a 'trace' of the page render.
// Process the trace to get the timing information.
async function getAuditResult(artifacts, resolveAuditResult) {
 const trace = artifacts.traces.defaultPass;

 const tabTrace = await artifacts.requestTraceOfTab(trace);
 const duration = extractDuration(tabTrace);

 // Tell Lighthouse the result
 resolveAuditResult({
 rawValue: duration,
 displayValue: duration,
 score: 1,
 scoreDisplayMode: 'manual',
 });
}

// Compute the duration of the specific User Timing measurement
function extractDuration(tabTrace) {
 let duration;

 // Find the User Timing measurement event
 const durationEvents = tabTrace.processEvents.filter(
 event =\> event.name === 'DIFFSET.RENDER.DURATION'
 );

 // Lighthouse records 'begin' and 'end' events for a measure
 const begin = durationEvents.find(event =\> event.ph.toLowerCase() === 'b');
 const end = durationEvents.find(event =\> event.ph.toLowerCase() === 'e');

 // Event duration in milliseconds
 duration = end.ts - begin.ts;
 duration = Math.ceil((end.ts - begin.ts) / 1000);

 console.info('\>\>\> Diff Rendering Only duration (ms): ', duration);

 return duration;
}

module.exports = DiffRenderingOnlyAudit;

Create a config file for Lighthouse that runs the above audit.

// example-config.js
module.exports = {
 extends: 'lighthouse:default',
 settings: {
 throttlingMethod: 'provided',
 onlyAudits: [
 'user-timings',
 'diff-rendering-only-audit',
 ],
 emulatedFormFactor: 'desktop',
 logLevel: 'info',
 },

 audits: [
 'diff-rendering-only-audit',
 ],

 categories: {
 pullRequestMetrics: {
 title: 'Pull Request Metrics',
 description: 'Metrics for pull request page',
 auditRefs: [
 { id: 'diff-rendering-only-audit', weight: 1 },
 ],
 },
 },
};

Run the above audit once.

lighthouse \<your url\> --config-path=./config.js --emulated-form-factor=desktop --chrome-flags="--headless"

Next, an example node script to batch run the Lighthouse audit.

const commandLineArgs = require('command-line-args');
const lighthouse = require('lighthouse');
const chromeLauncher = require('chrome-launcher');
const log = require('lighthouse-logger');
const { mean, sampleStandardDeviation } = require('simple-statistics');
const lighthouseConfig = require('./config.js');

// Command line arguments
const argsDefintion = [
 // n: number of runs to execute 
 { name: 'n', type: Number },

 // url: url to test against
 { name: 'url', type: String, defaultOption: true },
];
const args = commandLineArgs(argsDefintion);

if (!args.url) {
 console.error('No url specified');
 return;
}

const lighthouseFlags = {
 logLevel: 'info',
};

go();

async function go() {
 const n = args.n || 10;
 const diffRenderTimes = [];

 // Launch Chrome
 const chromeFlags = ['--headless'];
 const chrome = await chromeLauncher.launch({ chromeFlags });
 lighthouseFlags.port = chrome.port;

 // Run lighthouse n-times
 for (let i = 0; i \< n; i++) {
 console.log(`Start run ${i + 1}...`);
 diffRenderTimes.push(await runLighthouse());
 }

 // Compute statistics
 console.log('All render times:');
 console.log(diffRenderTimes.join('\n'));
 console.log('MEAN (ms):', mean(diffRenderTimes).toFixed(2));
 console.log('SD (ms):', sampleStandardDeviation(diffRenderTimes).toFixed(2));

 await chrome.kill();
}

async function runLighthouse() {
 const results = await lighthouse(args.url, lighthouseFlags, lighthouseConfig);
 delete results.artifacts;

 return results.lhr.audits['diff-rendering-only-audit'].rawValue;
}

Finally, run the batch script.

node batch.js \<your url\> --n=\<the number of executions\>

Running my new custom Lighthouse audit gave me these two sets of numbers (rendering times in milliseconds):

Source (lighthouse image): Wikimedia.org

So, the average time is lower by ~5%. Hooray!! All is well!

“But not so fast… the average isn’t a lot lower, and the rendering times are all a little different. How do you know this shows a real improvement?? And also, get back to the beer.”

You got me! And well spotted! This is the core of the question! The rendering times are all a bit different. I need to be confident the improved rendering time isn’t just a random quirk of this sample.

Luckily our friend Billy the beer brewer gave us a tool called the t-test. This is how Billy worked out which supply of hops would give him the best beer!

It lets me ask: Given the amount of ‘noise’ in my example rendering times, is this difference between the averages likely to be a real difference or just a random fluctuation? The phrasing you hear is: is it significant?

Using Billy’s tool, I get these two values:

t = 10.5457, p < .0001

This tells me: The probability that my improved rendering times came from random chance are less than one-in-ten-thousand 1 . I can conclude: the code change gives a speed improvement that averages only ~5%, but it’s highly likely to be a _real _5% **.

** Caveat: I’m glossing over a LOT of details around good experimental design. This is just highly simplified example! And FYI, results in the real, non-dev, world (medicine, etc.) are never as neat as in this example. They show a lot more noise.

“What’s the takeaway?”

This post is about opening developer eyes to the connection between statistics and coding. (It’s not meant to be a detailed how-to.)

The core message is: use the power beer gave us to make data-driven code decisions.

Many small improvements in performance can add up to big improvements — as long as those small improvements are real and not just random quirks. To make sure they’re real:

Collect metrics from your own code.
Run significance tests to see if you’re making a real difference.

“So, can I have a beer now?”

Yes.

This post was written by Christian Doan — “I’m a Senior Developer on Bitbucket frontend team. My current project is migrating legacy code to a modern SPA stack. I’m passionate about great user experiences and fast performance. Connect with me on linkedin.”

Originally published at bitbucket.org on February 26, 2019.

Preventing Git Rebase Fights

Ash Moosa — Thu, 11 Oct 2018 15:58:52 +0000

What’s a Git Rebase Fight?

Have you ever experienced this situation?

You go to merge your PR (pull-request), but the PR says it must be rebased before you can merge.

You rebase it, which kicks off a new build. But the build must complete before you’re allowed to merge.

The build completes! Yay! But while you were waiting, someone else managed to merge their PR, updating origin/master with their work. Uh oh!

You go back to step 1 and hope for better luck this time…

Bumping into this loop once or twice a month is not a big deal (especially if step 2 takes under 10 seconds). But sometimes the situation can become pathological.

Solution: Optimistic Build Status Propagation

You can use $(git diff TARGET...SOURCE | git patch-id) to prevent these rebase fights. This is handy when you know the build is very likely to succeed (e.g., squashes, amends, clean rebases, clean sync-merges).

The technique is called “Optimistic Build Status Propagation” because it uses the output of “git patch-id” as a heuristic to propagate build status to newer commit-ids without requiring the actual full build to finish. It works like this:

You push a new branch to your central git server.
The push triggers a build on your CI (continuous integration) server.
The build eventually succeeds. The branch’s tip commit is marked with a SUCCESS flag.
You decide to rebase your branch.
During the rebase your git server notices two things: 1. The commit before the rebase had a successful build filed against it, and 2. The rebase was clean (no conflicts).
The rebase triggers a build on your CI server. The CI server sends your git server an IN-PROGRESS notification.
Your git server receives the IN-PROGRESS notification. Because the git server also knows the rebase was clean, and also knows the pre-rebase commit had a SUCCESS flag, your git server optimistically marks the new tip commit with SUCCESS instead of IN-PROGRESS.

In other words, the “SUCCESS” flag from before the rebase is propagated to the commit created after the rebase. That’s why it’s called “Optimistic Build Status Propagation”. It lets you merge immediately after the rebase, since there’s no need to wait for the build to complete.

This optimistic window is temporary. Only the IN-PROGRESS flag is intercepted and replaced with a SUCCESS flag. Eventually the CI server will complete the rebased build and send a final SUCCESS or FAILURE notification. These are dutifully recorded of course, replacing any previous flags filed against the commit.

This technique is invaluable for shops running a fast-forward merge policy alongside a build-all-branches policy. If your builds are even just a little bit slow (e.g., 3 minutes or worse), your staff are probably waging an infinite rebase war against each other. Or they’ve found a better job somewhere else. Or they’ve disabled those merge policies.

If you’re on Bitbucket Server there is at least one plugin for this: PR-Booster for Bitbucket Server.

I’m not aware of any pre-baked solutions for this problem on Gitlab or Github.

What Causes Git Rebase Fights?

Fast-forward merge policy causes rebase fights.

A fast-forward merge policy only lets PRs merge if they are ahead of origin/master. In other words, PRs must be rebased before they can merge. The policy keeps git history neat, clean and linear by eliminating merge commits. But the policy can also cause rebase fights. I can think of two situations in particular where this happens:

High contention for the merge right-of-way and repo is so large that rebases are slow. I suspect this situation is rare and confined to very large teams. Rebases are only slow with very large repos, and you’d need at least 50+ engineers targeting the same origin/master before the contention would get high enough. Monorepos in particular may be vulnerable to this.
Slow secondary commit validation processes (e.g., builds must succeed before merge, but builds are 3+ minutes). I suspect this situation is much more common.

If your rebase fights are happening because of scenario 1 (very large repo + very large team), then you should probably forget about running a fast-forward policy. Sorry, but you need those merge commits. In exchange for a messier commit graph you get improved productivity. It’s a good tradeoff!

If your rebase fights are happening because of scenario 2 (slow secondary processes), then “Optimistic Build Status Propagation” is available as a solid mitigation. Under scenario 2 you can have both a clean commit graph and the productive team!

The rest of this blog post is about how $(git diff TARGET...SOURCE | git patch-id) works under the hood.

Triple-Dot-Diff and “git patch-id”

I refer to “git diff A…B” as triple-dot-diff. When people complain about Git’s usability, the triple-dot operator is certainly one of Git’s blemishes. The operator’s behaviour is inconsistent across various commands (e.g., “git log A…B” does something quite different).

The manual for “git diff” explains the triple-dot-diff like so:

“git diff A…B” is equivalent to “git diff $(git-merge-base A B) B”

Visually, it looks like this:

This is because the merge-base of master and branch is commit cc603d1, the last commit they had in common before they diverged. And so “git diff master…branch” is equivalent to “git diff cc603d1 ee7b565”.

Turns out clean rebases, squashes, merge-squashes, and sync-merges (and amends, of course) do not perturb this fundamental diff. The command “git merge…branch” (with three dots) is stable even if master advances or branch is rebased. The line numbers might change, and hunks might be rearranged, but the fundamental diff itself does not change unless there’s a conflict resolution (or an evil merge). Atlassian’s Auto Unapprove plugin explores this in detail in its issue #15.

If we were writing “Optimistic Build Status Propagation” from scratch, generating canonicalized diffs would be a big headache. Fortunately, the “git patch-id” command already has this covered, with some extra help coming from its “ — stable” option:

— stable

Use a “stable” sum of hashes as the patch ID. With this option

reordering file diffs that make up a patch does not affect the ID

Here’s some examples using master and branch from the diagram (clone from here if you must!):

git diff master...branch | git patch-id --stable
**790e0c0693c61e28fa1b3eea204bafe3946f5cba**

If I synch-merge (I’m on branch):

git merge master -m 'merge'
git diff master...branch | git patch-id --stable
**790e0c0693c61e28fa1b3eea204bafe3946f5cba**

If I retreat and rebase:

git reset --hard origin/branch
git rebase master
git diff master...branch | git patch-id --stable
**790e0c0693c61e28fa1b3eea204bafe3946f5cba**

The patch-id doesn’t change! This makes the command (triple-dot-diff piped into patch-id) perfect for determining when rebases and other common branch operations have not changed the underlying work sitting on the source branch. Since the underlying patch has not changed, one can optimistically presume the build will probably have the same result.

Conclusion

Fast-forward merges are great, because they avoid pointless merges and keep the history clean. Requiring successful builds before merging is great because it prevents broken builds. But add these together and you might find yourself in an infinite rebase fight!

Fortunately you can use $(git diff TARGET...SOURCE | git patch-id) to stop the fighting.

If you’re on Bitbucket Server, install the PR-Booster add-on to deploy the fix instantly.

Otherwise roll your own, and let me know when you do! Email me at julius at mergebase.com.

Happy rebasing!

(p.s. For those on Bitbucket Server, I use Control Freak to enforce a fast-forward merge policy on git repositories I control.)

This is a guest post for Bitbucket written by Julius Musseau at mergebase.com. To contribute to the Bitbucket blog, apply here.