DEV Community: Omar Bahareth

Are there functions similar to Ruby's `dig` in other languages?

Omar Bahareth — Thu, 22 Aug 2019 15:18:26 +0000

I really like using Ruby's Array#dig and Hash#dig operator (introduced in Ruby 2.3) to quickly and safely access deeply nested structures

I would love to see different versions of it in other languages so I can use them more too!

Here's how dig works:

Assuming we have an orders array that looks like this.

orders = [
  {
    id: 1,
    customer: {
      name: "Customer 1",
      phone: "1234"
    }
  },

  {
    id: 2
  },

# ...
]

We can easily navigate through this structure with dig like so

orders.dig(0, :customer, :phone) #=> "1234"

We can also not worry about any of the "in-between" objects not existing, as it will return nil as soon it finds something that doesn't exist.

orders.dig(1, :customer, :phone) #=> nil

It returns nil the moment it finds that customer doesn't exist, and it makes me avoid checking if keys exist every time I want to access a nested object.

What are some cool ways to access nested data like this in other languages? I ask because I want to learn and because I probably do it in overly-convoluted ways at the moment.

Thanks for reading!

Engineering Management: An Overview

Omar Bahareth — Sun, 16 Jun 2019 20:07:21 +0000

I was an engineer for most of my career and one day I became a manager and I was quite lost and afraid in the beginning, I realized I hadn’t shared my experiences from back then. I’ve been doing engineering management on and off for the past couple of years and I decided to share a short summary of what I learned.

A lot of this is stuff I learned from books, articles, websites, and some of this is stuff I learned on the job. Full credit to all the resources I learned from is at the bottom in the “Great resources” section, and I heavily recommend exploring them if this article captured your interest. I like to use the term “Maker” (engineer, designer, or anyone who makes stuff) and “Manager”, both of which I learned from the book Leading Snowflakes. I wrote this from the perspective of someone who has been a Maker for most of his career and transitioned to Management a couple of years ago.

What to expect

You need to write, a lot

Meeting notes, peer feedback, feedback to you, documenting a task before you hand it to an engineer, and so on. Think of yourself as the standard for documentation in your organization or team, because you kind of represent that now. You’ll get so much in your plate that it’s impossible to remember it all, writing is your best friend.

You’ll talk to people and be in meetings a lot more

Management is all about people interactions, you’ll be dealing with people a lot more than before and that means meetings. Too many meetings for your team is a productivity killer and that’s also something you’ll need to figure out how to cut down if it becomes an issue.

Learn to measure your success by your team’s success, not what you make yourself.

When I started being a leader I was used to measuring myself as an engineer or as a “maker” (measuring by what I make). When I first started being a manager I used to feel like I haven’t accomplished anything in my day and this was difficult.

You’ll be exposed to a lot more conflicts

You’ll see a lot of conflicts between people, sometimes over very small issues. A lot of your time will go towards handling them. This may not be the case for every manager but it was the hardest part of the job for me.

Hard Choices

You’re gonna have to let some people go, you’re gonna have to step in to break up conflicts, and sometimes you’ll have people who are trying their best but still hurting teammates. This isn’t an easy path and it’s full of people issues and conflicts. Even if you have a great hiring process, it won’t be perfect and you’ll hire the wrong person sometimes. There’s no right or wrong answer in almost any situation, you need to have peers and mentors you can trust and ask for guidance. Don’t make hard decisions alone, and definitely don’t make them when you’re not calm.

You’ll need to do things that many people don’t want or aren’t willing to do

Improving code standards, bug report structures, doing some things manually until they’re automated, improving communication between your team and other departments, and so on. Your goal is to keep the team focused and removing obstacles from their way, and those obstacles are often easy/boring tasks that no one wants to do.

What to strive to do

1:1 meetings

Regularly sit down with your team mates, connect with them, understand their goals and what they want to achieve and learn, understand what’s troubling them. Put this on a schedule, it’s extremely important to do this regularly. Write down the main points that go on in your 1:1 meetings, it’s important for everything you need to do on this job.

Mentorship

You need to help your teammates get better, sometimes in what they want to get better at, and sometimes in things you believe they need to learn for the sake of their team. This can be accomplished by pairing them with someone more experienced in an area they want to get better at, getting them access to websites like PluralSight, etc. Sometimes by mentoring them yourself. You have to give them goals, ask them to experiment with new things, help them write articles, give talks, and so on.

Scheduling

You’ll probably be in a lot of meetings and people need an easy way to sit down and talk with you. Have an easy way for them to book some of your time, Google Calendar, Freebusy.io, Calendly, etc. are all great tools for this.

You also need to learn to switch between your Maker and Manager modes, it’s good to regularly have this scheduled either some days of the week or some hours of certain days. You need time to hop on in and review some work (pull requests, designs, etc.) and contribute to them too.

Keeping your “Maker” skills polished

You won’t have as much time for “Making” as you used to before, and you might get rusty. Take the time to keep yourself up to date with what your team is doing, and of new developments in your field. You need to be able to understand at a general level what your team does so you can help them prioritize.

Keep a list of tasks

Don’t forget anything. Write things down. Anything anyone asks of you, jot it down on some app and slap a due date on it (I use Todoist). This list is also great for delegating. Got a complex task coming up that you’d like a teammate to have experience in? Delegate it. Got a lot of meetings today and your teammate asked for access to something or to try out a new tool, write down the task, don’t forget them.

Hiring

The first thing you need to look at when hiring is whether these candidates will fit in with your team or not. Sit down with your teammates and peers and decide what’s a must-have and what’s a nice-to-have for your workplace. Do you care about about writing well in your organization? Are you ok with people learning it on the job? Or is it a must-have to even join?

Avoid brain teaser interview questions, avoid theoretical scenarios and whiteboard questions that people will never have to do on the job, and make sure your hiring process involves the most real scenarios as possible. Get the candidate to work with your team on an actual task, and compensate them for it. Microsoft’s new process is my favorite way to hire, I highly recommend reading it. Not all organizations have the resources to do all this stuff and that’s understandable but try your hardest to accomplish these two things when hiring:

Get the candidate to do some real work.
Have your teammates and the candidate interact when doing real work.

Hire juniors

Too often organizations only look for people who already know most of what they need and don’t hire enough juniors. Juniors are great to have in any organization because they want to learn and there’s a lot of them. A lot of the time after juniors grow and gain more skills within your organization, they’ll start to get better offers from other places so some think this isn’t a good investment, but I believe this is something organizations must try their hardest to do, especially in developing countries, because those juniors need to grow and they need help and training, and because you’ll have more manpower to deal with the many small bugs that are often overlooked.

Let your team grow / delegate

As a manager, you need to delegate. Jumping into Maker mode to get something done is something you were probably used to before but you now need to let your team be responsible for most of the “Making”.

Learning by doing is the best way to learn. If there’s an area you’re an expert in as a Maker, don’t immediately jump in to Maker mode and try to do it, let your teammates experience it and grow. Write them a short guide of what needs to be done and things to look out for and trust them to get it done.

Encourage knowledge sharing

Ensure your teammates have an accessible place to find and share lessons learned, no matter how small they are (Something like StackOverflow for teams, etc.). Ensure that nobody on your team is afraid or embarrassed to ask questions, regularly encourage your team to ask anything they can think of, and make sure your team is composed of people who are ok with that.

Always be open for feedback and change

I like to have a GitHub repo in my organization for everything related to how I do management, how 1:1 meetings are run, what I strive to do as a manager, what teammates can expect from me, what I expect from them, and so on. I put it in a GitHub repo because engineers are generally used to working with GitHub and are used to changing things with pull requests, so I wanted to be like that. Make a pull request to change how your manager does something 😛

Get your team to be able to function without you

This isn’t to say that you’re supposed to be useless or make yourself obsolete but that you should not be a blocker for your team. Imagine yourself going on a vacation for a month, what would go wrong? What tasks are you a blocker for? Will the person leading in your stead know what to do? Have you left them the necessary information? Does the team know what to work on? Do they know what to do or who to talk to if some disaster happens? Try going on short vacations every once in a while and see what goes wrong. At the beginning you’ll most likely find that a lot of the things you handled as a “Maker” aren’t being done anymore and you need to both make the team aware of them and delegate them.

Great resources

Books

Leading Snowflakes — Full of practical advice on day-to-day things you need to do as a manger.

The Manager’s Path — A guide for tech managers going through increasing levels of management complexity.

High Output Management — Great book on running and scaling a company.

Websites

Manager Readme — A community of sharing your “manager guides” with other managers.

The Watercooler — A community for leaders looking to become better.

Articles

Google's 10 year research on what makes a good manager.

Rethinking how we interview in Microsoft’s Developer Division — Great article on hiring processes.

Undervalued Software Engineering Skills: Writing Well.

Newsletters

Software Lead Weekly — A weekly email for busy people who care about people, culture and leadership.

I’m not much of a video/audio person so I don’t have great resources for those mediums, if you have recommendations for more resources, share them below!

I just moved back to development from being Head of Engineering, Ask Me Anything!

Omar Bahareth — Tue, 09 Apr 2019 11:23:34 +0000

My Mac Setup

Omar Bahareth — Sun, 30 Dec 2018 11:10:48 +0000

This post is my setup by the end of 2018, I keep my-mac-os repository on GitHub up to date with my latest setup.

Hardware

Computer: 15" Late 2016 Macbook Pro with Touch bar.

Trackpad: Apple Magic Trackpad 2.

Monitor: BenQ GW2765HT 27-Inch.

Keyboards:

ErgoDox EZ Shine for work and programming.
Microsoft Universal Foldable Keyboard for typing on the go.
Filco Majestouch Ninja 2 for gaming.

Applications

I'm slowly transitioning to cross-platform software because I want to be able to use the majority of my apps on any OS.

Productivity

Alfred - Launcher

Alfred is both my launcher, clipboard history manager, and entry point to lots of awesome workflows.

It has a great community and amazing workflows that you can use.

Here's a small example of me using it to quickly run/evaluate Ruby code:

Todoist - Simple and quick task manager

Todoist helps me plan out everything I want to do either for the short term or long term (I'm not sure if I follow GTD).
It also has global quick add with a hotkey. Together with lists, priorities, powerful search and a lot more.

Trello - Project management tool

Simple project management tool with tons of great integrations.

Pipefy - Automation on boards and forms

When I need something like Trello but with a lot more automation/power (e.g. having a public form, sending emails when things move to a certain column, triggering other actions, etc.)

1Password - Password manager

Generate all of my passwords with it and keep everything in a secured and encrypted vault kept secure by a master password and two factor authentication.
No longer need to remember passwords and I now have a unique password for every website that I am signed up on whilst activating two factor authentication wherever possible.

MindNode - Interactive Mind Mapping

I write A LOT, too much actually, MindNode helps me write short summaries of my thoughts that I "connect the dots" (or lines) between.

PDF Expert - PDF reader/editor

Super fast PDF viewer.
Best experience for editing PDFs or filling PDF forms.

Contexts - Window switcher

Allows me to fuzzy search through all the currently active windows that I have.

Makes jumping to the right window I need effortless. I often may have many VS Code instances with different projects running and this lets me switch to the project I need in seconds.

Bartender - Menu bar organizer

Allows you to customize and hide the contents of your menu bar and improve the aesthetics of your OS. Here is how mine looks:

Magnet - Organize your workspace

Lets me use keyboard shortcuts (or more rarely, drag-drop to edges) to tile windows on my screen.
Keyboard shortcuts to move windows across monitors.

Yoink - Simplify and Improve Drag and Drop on your Mac

A small/quick utility I use to drop files from multiple locations/apps into to take an action on them later.

Little Snitch - Control incoming/outgoing network traffic

Amazing networking tool that gives you a clear picture of what connections are incoming to your computer and what are outgoing.
Takes a bit of time to set it up correctly and is quite an insightful experience first turning it on and having it notify every couple of seconds that some app is trying to send data to some server and whether you want to allow that.
This is essential if you want to take control of what information gets sent out from your computer and what connections have right to connect to your data.

Code

Visual Studio Code - Code editor

My favorite editor that I use to write code in. I use many extensions for it.
I use the Nord theme, Material Icon Theme and the Fira Code font.

Here is how it looks:

Neovim - Powerful text editor

Opening, editing, modifying, and searching through huge files.
Go-to commandline text editor.
I use vim-plug to manage plugins.

Hyper - Terminal Emulator

I use Fish as my shell together with Fisher to install Fish plugins I use.
You can find my dotfiles here.
I use the Pure prompt and the Nord theme.

Dash - API Documentation Browser

Allows you to download any docset that you might want to use, search for any method, class or anything that you need very quickly, comes with the amazing Alfred workflow to simplify the process of searching for the right things.

GitKraken - Git client

Great cross-platform Git client that integrates well with GitHub, BitBucket, GitLab, and supports GitFlow.

Kaleidoscope - Powerful diff tool

Really great, precise, and beautiful diffing.
Great at viewing/merging conflicts.
Can do directory and image diffs as well.

Gitify - GitHub issue & pull request notifiations

Neat little app that sits in the menu bar and lights up when I have GitHub notifications.
Can view all notifications and mark them as read.

Social

Textual - IRC Client

The best macOS IRC client I've found.

Rambox - Open Source Multi-messaging App

A single app I use for chatting with people on:
- Telegram.
- WhatsApp.
- Messenger.
- Discord.
- Slack (sometimes).

Spark - Email client

Love how it smartly categorizes emails by categories.
I approach all of my email tasks in GTD style. Keeping my email Inbox close to 0 at all times.
I love being able to easily take batch actions on emails.

Writing

Day One - Digital journal

Day One is my digital life journal.
If you don't journal, I suggest you to start, it is a very powerful mind cleanser and acts as a wonderful history record of your life.

Byword - Distraction-free Markdown Editor

A great minimal distraction-free markdown editor, I use it when I want absolute focus.

Quiver - The Programmer's Notebook

Powerful cell-based editor that operates based on "cells" (e.g. markdown cell, code cell, LaTeX cell, etc.) for mixing and matching different kinds of formats.
Plain JSON data format.
Cloud syncing.
Charts.

Evernote - Large writing, scans, web clips, and annotations

I do my huge writing here.
If I want to access a document from anywhere, I usually store it here.

Music

Google Play Music - Music Streaming

My goto music streaming solution.
Easiest for me to setup from where I live (Saudi Arabia).

YouTube Music - Music Streaming 2nd solution

Google Play Music will most likely be merged with this, so it might be best to get used to it.
I use it on the rare occasion that I don't find something on Google Play Music.

Images

Google Photos - Cloud storage for photos

Easiest way to backup my photos and access them from multiple places.
Machine learning, auto detects images.
😖 Kind of scary.

Utilities

xScope - Measure. Inspect. Test.

A powerful set of tools for Mac OS X that are ideal for measuring, aligning and inspecting on-screen graphics and layouts.

iStat Menus - Mac system monitoring from menu bar

Great system monitoring tools which I use to quickly see my CPU/network/memory usage.
Has a great calendar popup that shows when you click date/time in the menubar. This is something I was really used to from Windows/Linux that I'm glad to have back again.

HyperDock - Window Previews for Dock

Window previews (thumbnails) when hovering over applications on the dock.
Calendar previews when hovering over the Calendar.

Paw - Great API tool

Full-featured HTTP client that lets you test and describe the APIs you build or consume.

Browsers

Google Chrome

My main web browser because it easily syncs my extensions across machines.
I use these extensions and this theme.

Safari

Incredibly fast/lightweight web browser.
Since I don't have extensions on it, I use it for when my Chrome extensions break sites.

Command Line Apps

autojump - A cd command that learns, easily navigate directories from the command line.
asdf - One version manager for all my programming languages.
- elixir
- golang
- java
- nodejs
- redis
- ruby
- rust
fzf - Command-line fuzzy finder.
colorls - Beautify ls command with color and font-awesome icons.
bat - Cat clone with wings.
git - Version control.
curl - Transfer data from or to a server.
htop - Interactive text-mode process viewer for Unix systems.
httpie - HTTP client.
curl - Transfer data, supports various protocols.
howdoi - Instant coding answers.
asciinema - Terminal session recorder.
tldr - Simplified and community-driven man pages.
now - Real time global deployments served over HTTP/2.
yarn - Fast, reliable, and secure dependency management.
hub - GitHub wrapper.
xsv - Fast CSV command line toolkit written in Rust.

Lessons Learned From a Year of Fighting With Webpack and Babel

Omar Bahareth — Tue, 25 Dec 2018 21:08:08 +0000

I've been using React for more than a year now and a lot of the time things broke, slowed down our build speeds, and didn't work in the browsers I was transpiling for. I decided to share the lessons I've learned in the hopes they might help others out.

Be very careful with importing large libraries

Trying to compile large libraries (like react-plotly or PDF libraries) can take your Webpack compile from seconds to 10 minutes+. If a package is slowing down your compile, consider using a CDN version. We simply used script tags, but there are Webpack plugins that can help with that too:

Try to find a webpack plugin for your dependencies

Just importing packages like moment.js or lodash brings in a lot of bloat that you probably don't need. Try to import what you need only, or better yet find a webpack plugin that removes the unused things from your bundle, because selective imports don't always work. As one example, there's a webpack plugin that removes a lot of the unnecessary bloat added by Moment.js.

Google actually has a nice repository listing some common problematic dependencies.

Inspect your bundle with Webpack bundle analyzer

Webpack Bundle Analyzer is extremely helpful to see what exactly is going into your bundle. In the screenshot above, you'll notice that moment.js has lots of localization files that your app probably doesn't need. Webpack Bundle Analyzer can help you easily spot these issues.

Add es-check to your CI pipeline early on

es-check will help you find out which ES version your bundle is using, it's super useful to find out if you're somehow suddenly not producing ES5 anymore. Even if you're using Babel and browserslist, you might be importing a node module that's not even meant to be used in browsers, or even a package that's not being distributed as ES5. Add es-check to your continuous integration pipeline early on and it should help you find out if your bundle ever stops working with ES5, and that'll help you find which package is the culprit so you can then transpile it.

Transpiling a node_module

We had imported a very simple package called hex-rgb that's not even meant for browsers and this tiny package made our bundle not ES5-compatible anymore. Such packages should go through Babel and be transpiled.

In your webpack config, your babel loader's exclude field probably looks like this: /node_modules/ . We need to make a regex that excludes node_modules except the specific ones that should be transpiled:

// Exclude all node modules except hex-rgb and another-package
/node_modules\/(?![hex\-rgb|another\-package])/

And once again, this might not be a good solution for large packages as it can drastically slow your build time and you might want to switch to a CDN version instead.

Follow this issue from the babel-loader repo to stay up to date on how to handle cases like this.

Update (01 Feb 2018)
I published a package called are-you-es5 to help out more with this. It gives you easy to read output, and even builds the regex you need for transpiling those specific modules, check it out!

Use Browserslist to specify your target browsers

Browserslist lets you specify which browsers to transpile for.

> 1%
ie >= 8

This simple configuration targets browsers with usage more than 1% global usage, and IE versions 8 and above.

Use babel.config.js over .babelrc (for Babel ≥ 7.0)

Favor using babel.config.js to configure Babel over .babelrc. If you want to transpile node_modules (which is now becoming a very common case with webapps), then you should use babel.config.js.
.babelrc can be overridden by another .babelrc belonging to a node_module that you're transpiling and that can lead to all sorts of weird issues.

Make your webpack-dev-server logging output friendlier

Change your webpack-dev-server config to this

devServer: {
  noInfo: true,
  stats: 'minimal'
}

Add WebpackBar to get much less-verbose, friendlier, and more concise output.

Note: The first configuration is meant to be combined with Webpack Bundle Analyzer, as it suppresses console output for things related to your bundle that Webpack Bundle Analyzer already shows. If you're not using Webpack Bundle Analyzer, don't apply the first step.

What I hope to see

We need a lot of improvements in our tooling around this process. Most webpack and Babel configs exclude the node_modules folder (with good reason, compilation would take an insanely long time if they didn't!), but I personally feel this is an area where we need smarter tools. It would be quite convenient if everything "just worked" only by configuring Webpack, and Babel along with browserslist without having to dig through dependencies and the often unique problems they introduce to our build pipeline ourselves. The package.json file could be a good starting point towards this goal. There are a couple of things that if used could help tools determine if a package should be tranpiled or not:

The browser field, this should be used instead of the main field if your package targets browsers. Meaning that we should at least get a warning if we specify that we target browsers in our package.json and we import a package that doesn't have the browser field in its package.json.
The engines field, This is usually used to specify the node versions your package targets, but I honestly think it should also be used to specify ES versions, since a lot of NPM's usage is targeting browsers nowadays. If a package doesn't have any of these fields and we specified that we target browsers and ES5 then we should:
- Get a warning when we import it.
- Have Webpack plugins that detect these packages and un-exclude them so they get transpiled automatically.

Closing Notes

Webpack and Babel are awesome, this article isn't trying to say otherwise. We wouldn't be able to use a lot of things if it weren't for them, but the experience of using them together needs to get better. I faced many of these issues over the course of many months and understanding them/finding solutions was incredibly difficult (error messages don't exactly tell you what went wrong and searching for them doesn't always give relevant results), and I hope this article can act as the guide I had hoped to find back then.

A Quick Dive Into GraphQL, Part 1: Building a GraphQL API

Omar Bahareth — Fri, 16 Nov 2018 12:27:52 +0000

What is GraphQL?

“A query language for your API.”

Think of it as being able to send queries to an API endpoint to fetch exactly what you want, rather than the traditional REST way of having many endpoints per resource. The way that it personally clicked for me is thinking of it as

Take a look at this example from the GraphQL website to get an idea of how it works.

request (GraphQL Query) is at the top, response is at the bottom

As an API provider, GraphQL lets you do this by letting you define a schema and how to fetch the resources in it.

As an API consumer, all you have to do is build the query to fetch exactly what you need.

Who is using it?

GraphQL just got its own foundation, and you can see a full list showing who’s using it from their website. Below are some notable articles from companies who’ve used GraphQL where they explain why they used it and their experiences with it:

Use cases

“Ask for what you need, get exactly that.”

Sometimes you just want to get a person’s name, other times you want to get the person along with his name and date of birth.

“Get many resources in a single request.”

Other times you might want to get the person, their home town, along with their friends and their home towns too. With other solutions, you’d usually either have to make different routes for each case, or write a way to request specific parameters yourself.

I see these two features as extremely useful for the frontend web development (particularly with frameworks like React, Vue, Angular, etc.) and mobile apps.

Another more advanced feature that I’ll cover in a later post is stitching, which lets you combine multiple schemas (from different APIs) into one; meaning you can fetch resources from different APIs with a single request.

Consuming a GraphQL API

There’s a GraphQL “IDE” that you can use to explore GraphQL APIs called GraphiQL. They have a demo playground here with Star Wars films info that I used to construct the next example. The real cool thing about GraphiQL is that you can just keep hitting CTRL + Space like you would in any IDE, and you’ll get intelligent autocomplete. If you’re currently within a block related to films, you’ll see film fields that you can query for. This really helps explore and try out GraphQL APIs quickly and easily. Here’s an example query that shows some of the power of GraphQL.

{
  # The first 2 films, after a given cursor value
  allFilms(first: 2, after: "YXJyYXljb25uZWN0aW9uOjA=") {

    # An edge is an entry in a "list"
    edges {

      # Get a cursor value for each "edge"
      # Used for pagination, in combination with filters like
      # before and after.
      cursor

      # Each edge has a node, where the node has
      # the data we're interested in
      node {
        # Film title, release date, and director
        title
        releaseDate
        director

        # The first 2 species that appeared in this film
        speciesConnection(first: 2) {
          edges {
            node {
              # The name of the species
              name
            }
          }
        }
      }
    }
  }
}

This query gets the first 2 films (after a specific film’s cursor value) with their title, release date, and director. It also gets the names of the first 2 species that appeared in this film. We just fetched multiple resources with a single request.

Here’s what the result looks like

{
  "data": {
    "allFilms": {
      "edges": [
        {
          "cursor": "YXJyYXljb25uZWN0aW9uOjE=",
          "node": {
            "title": "The Empire Strikes Back",
            "releaseDate": "1980-05-17",
            "director": "Irvin Kershner",
            "speciesConnection": {
              "edges": [
                {
                  "node": {
                    "name": "Human"
                  }
                },
                {
                  "node": {
                    "name": "Droid"
                  }
                }
              ]
            }
          }
        },
        {
          "cursor": "YXJyYXljb25uZWN0aW9uOjI=",
          "node": {
            "title": "Return of the Jedi",
            "releaseDate": "1983-05-25",
            "director": "Richard Marquand",
            "speciesConnection": {
              "edges": [
                {
                  "node": {
                    "name": "Human"
                  }
                },
                {
                  "node": {
                    "name": "Droid"
                  }
                }
              ]
            }
          }
        }
      ]
    }
  }
}

Services and Tools

Gatsby

Gatsby is a new framework that’s rapidly rising in popularity (and very well documented), it’s static PWA (Progressive WebApp) generator that lets you build websites using React, and fetch data using GraphQL. Even though you’re deploying “static” files, you can build e-commerce solutions and CMS and just do a whole lot of stuff that you might not think is possible with a static website.

Gatsby abstracts things away via GraphQL. You want to fetch a file from your disk? You want to fetch cards, boards, or lists from Trello? You can do all of that just by sending GraphQL queries; usually through existing plugins that have already built the GraphQL API for you. It even comes with the GraphiQL IDE mentioned above and you can try fetching live data instantly, before you write even a single line of code.

I’ve personally had a very rapid and rich development experience with Gatsby; especially when interacting with the community, but I did struggle with one thing: localization. Localization can mean different things depending on who you ask. It could be just translating strings, different routes per locale, or a whole lot more. In my case I wanted to translate strings and style content differently (right to left, for Arabic) and while I always found a “Gatsby way” of doing things, this was on area where I had to fiddle around a lot but I did end up having something that worked well after some trial and error.

Hasura

Hasura instantly and “automagically” builds a GraphQL API out of your PostgreSQL database (you can quickly choose what is exposed through the API and who has access to it); which was of course a very logical development, since both SQL and GraphQL use schemas. Hasura even goes beyondjust making a GraphQl API for you, it provides subscriptions to database changes, live queries, triggers when database changes happen, and a whole lot more.

Apollo

Apollo is probably one of the friendliest ways to deal with or adopt GraphQL. In the past, they used to offer a client (React, JS, iOS, and Android), engine (for tracing and error tracking), and a server that helped you incrementally adopt GraphQL (even tools to help translate your REST API to GraphQL). Their offerings have great DX and they have dev tools and plugins for most editors, and all ofthat is open source. It really feels like they’re covering the GraphQL experience from A to Z.

They recently launched the Apollo Platform which combines their open source tools with cloud services

How do I use it?

I haven’t used GraphQL extensively yet, I’ve only used it in one of my projects, which is a website (work in progress) for listing tech conferences and meetups in the MENA region, and it’s built using Gatsby.

I also like to explore GraphQL APIs I come across using Insomnia, because it has similar features to GraphiQL.

Where do I get started?

Official Introduction to GraphQL
Libraries for clients.
Awesome GraphQL - This is a great list that should help you find anything you’re looking for.

Closing Notes

The comparison between GraphQL and REST is always made so I’ll briefly add my opinion here. I see them both as different tools in your toolbox. As always choose the right tools for your needs and use cases, and most importantly choose something you and your team are familiar and comfortable with, and areable to stay productive in.

Stay tuned for the upcoming Part 2, “Building a GraphQL API”.

Indexing PDF For Searching Using Tika, Nokogiri, and Algolia

Omar Bahareth — Sun, 02 Sep 2018 17:40:37 +0000

Mirror of this Medium post.

Figuring out where to start from

This is the first guide I’ve written so bear with me, and please provide feedback!

I was working on a project that required me to have some really powerful search capabilities that work for multiple languages, and especially searching through file contents (I initially started with PDF). I needed something working and I didn’t have a lot of time. Usually a quick Google search would turn up a tutorial or guide on how to tackle a task like this, but this time I didn’t have any luck, so I decided I’d share my experience after I had something working.

I went ahead and looked at StackShare and found a category for Search as a Service, I had always heard amazing things about Elastic Search and I checked out most of the other tools available there and contacted their support teams asking for some guidance. I was surprised to get a response only two minutes later (literally) from Adam Surak, director of infrastructure at Algolia (a search service used by Medium surprisingly, and even Twitch) and he helped me out quite a bit. He directed me towards Apache Tika which as their page states:

The Apache Tika™ toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF). All of these file types can be parsed through a single interface, making Tika useful for search engine indexing, content analysis, translation, and much more.
Adam also recommended that I split the PDF into paragraphs because it assures that the searched text isn’t too long and that the relevancy is high.

Extracting PDF Contents

I was able to quickly and easily acquire Tika via HomeBrew:

brew install tika

I’m going to use a PDF version of Dracula (and then I cut out everything but the first 3 pages) that I acquired for free from PDFPlanet and I ran:

tika -r dracula-shortened.pdf > dracula.html

The -r option is for pretty printing, and Tika outputs HTML for easy parsing (I used > to pass the HTML output into a file called dracula.html). Let’s look at the metadata that we got out of that:

<head>
<meta name="date" content="2017-01-05T18:08:24Z"/>
<meta name="pdf:PDFVersion" content="1.7"/>
<meta name="xmp:CreatorTool" content="Acrobat PDFMaker 5.0 for Word"/>
<meta name="Keywords" content="free, PDF, ebook, ebooks, Planet PDF, download, classic, classics"/>
<meta name="access_permission:modify_annotations" content="true"/>
<meta name="access_permission:can_print_degraded" content="true"/>
<meta name="subject" content="Free Planet PDF eBooks -- an assortment of some of the most popular classics. Free!"/>
<meta name="dc:creator" content="Bram Stoker"/>
<meta name="dcterms:created" content="2002-09-20T06:10:21Z"/>
<meta name="Last-Modified" content="2017-01-05T18:08:24Z"/>
<meta name="dcterms:modified" content="2017-01-05T18:08:24Z"/>
<meta name="dc:format" content="application/pdf; version=1.7"/>
<meta name="Last-Save-Date" content="2017-01-05T18:08:24Z"/>
<meta name="access_permission:fill_in_form" content="true"/>
<meta name="meta:save-date" content="2017-01-05T18:08:24Z"/>
<meta name="pdf:encrypted" content="false"/>
<meta name="dc:title" content="Dracula"/>
<meta name="modified" content="2017-01-05T18:08:24Z"/>
<meta name="cp:subject" content="Free Planet PDF eBooks -- an assortment of some of the most popular classics. Free!"/>
<meta name="Content-Length" content="48741"/>
<meta name="Content-Type" content="application/pdf"/>
<meta name="X-Parsed-By" content="org.apache.tika.parser.DefaultParser"/>
<meta name="X-Parsed-By" content="org.apache.tika.parser.pdf.PDFParser"/>
<meta name="creator" content="Bram Stoker"/>
<meta name="meta:author" content="Bram Stoker"/>
<meta name="dc:subject" content="free, PDF, ebook, ebooks, Planet PDF, download, classic, classics"/>
<meta name="meta:creation-date" content="2002-09-20T06:10:21Z"/>
<meta name="created" content="Fri Sep 20 09:10:21 AST 2002"/>
<meta name="access_permission:extract_for_accessibility" content="true"/>
<meta name="access_permission:assemble_document" content="true"/>
<meta name="xmpTPg:NPages" content="3"/>
<meta name="Creation-Date" content="2002-09-20T06:10:21Z"/>
<meta name="resourceName" content="Dracula_NT-shortened.pdf"/>
<meta name="access_permission:extract_content" content="true"/>
<meta name="access_permission:can_print" content="true"/>
<meta name="meta:keyword" content="free, PDF, ebook, ebooks, Planet PDF, download, classic, classics"/>
<meta name="Author" content="Bram Stoker"/>
<meta name="producer" content="Acrobat Distiller 5.0.5 (Windows)"/>
<meta name="access_permission:can_modify" content="true"/>
<title>Dracula</title>
</head>

There’s a lot of useful data there, and you can use Tika to get metadata, detect content language, and so many other powerful options, but in our case we are more interested in the body, here’s a snippet of what the body looks like (2nd page of the PDF):

<body>

<div class="page">
   <p/>
   <p>Dracula
   </p>
   <p>2 of 684
   </p>
   <p>Chapter 1
   </p>
   <p>Jonathan Harker’s Journal
      3 May. Bistritz.—Left Munich at 8:35 P.M., on 1st
   </p>
   <p>May, arriving at Vienna early next morning; should have
      arrived at 6:46, but train was an hour late. Buda-Pesth
      seems a wonderful place, from the glimpse which I got of
      it from the train and the little I could walk through the
      streets. I feared to go very far from the station, as we had
      arrived late and would start as near the correct time as
      possible.
   </p>
   <p>The impression I had was that we were leaving the
      West and entering the East; the most western of splendid
      bridges over the Danube, which is here of noble width
      and depth, took us among the traditions of Turkish rule.
   </p>
   <p>We left in pretty good time, and came after nightfall to
      Klausenburgh. Here I stopped for the night at the Hotel
      Royale. I had for dinner, or rather supper, a chicken done
      up some way with red pepper, which was very good but
      thirsty. (Mem. get recipe for Mina.) I asked the waiter,
      and he said it was called ‘paprika hendl,’ and that, as it was
      a national dish, I should be able to get it anywhere along
      the Carpathians. </p>
   <p/>
   <div class="annotation">
      <a href="http://www.planetpdf.com/mainpage.asp?eb"/>
   </div>
</div>

And here is the 2nd page in the PDF for comparison:

From the output we can see that each page is a div having page as its class, and then we have each paragraph as the inner text of a paragraph element p

This is very nicely formatted really easy to parse.

Using Tika in Ruby

For me, the next step was to use Tika in Ruby (because my stack relies on Ruby on Rails). Some quick searching lead me to Yomu and using it in Ruby became as easy as:

require "yomu"
yomu = Yomu.new "dracula-shortened.pdf"
puts yomu.html

Parsing HTML Using Nokogiri

I have used Nokogiri in the past to parse HTML and it’s pretty easy in this case.

require "nokogiri"
require "yomu"

def invalid_paragraph?(str)
  disallowed_strings = [ "", " ", "\n", " \n" ]
  disallowed_strings.include?(str)
end

def get_pdf_paragraphs(filename)
  yomu = Yomu.new(filename)
  paragraphs = []
  doc = Nokogiri::HTML(yomu.html)
  page = 0

  doc.css('.page').each do |node|
    node.css('p').each do |paragraph|
      paragraph_text = paragraph.inner_text

      next if invalid_paragraph?(paragraph_text)

      paragraphs << { text: paragraph_text, page: page }
    end

    page += 1
  end

  paragraphs
end

paragraphs = get_pdf_paragraphs("dracula-shortened.pdf")

I use a CSS selector to loop through anything with page as its class, and then I loop through all the paragraph elements p within that page, and I’m just incrementing the page variable (I do this because knowing which page the text was found on is important for my use case). I had a bunch of empty string paragraphs, paragraphs there were just a single space character or a newline character, and a space followed by a new line, so I added a simple function invalid_paragraph? to easily skip those pieces of text (I could’ve written a regular expression but it didn’t seem worth it in this case, especially since I wanted an easily readable tutorial).

Let’s take a look at paragraphs (I cut out a good chunk of the PDFs before pasting this output so I don’t have a really gigantic example):

[
    {

        :text => "Dracula \nBram Stoker \n",
        :page => 0
    },

    {
        :text => " \nThis eBook is designed and published by Planet PDF. For more free \neBooks visit our Web site at http://www.planetpdf.com/.",
        :page => 0
    },

    {
        :text => "Dracula \n",
        :page => 1
    },

    {
        :text => "2 of 684 \n",
        :page => 1
    },

    {
        :text => "Chapter 1 \n",
        :page => 1
    },

    {
        :text => "Jonathan Harker’s Journal \n3 May. Bistritz.—Left Munich at 8:35 P.M., on 1st \n",
        :page => 1
    },

    {
        :text => "May, arriving at Vienna early next morning; should have \narrived at 6:46, but train was an hour late. Buda-Pesth \nseems a wonderful place, from the glimpse which I got of \nit from the train and the little I could walk through the \nstreets. I feared to go very far from the station, as we had \narrived late and would start as near the correct time as \npossible. \n",
        :page => 1
    },

    {
        :text => "The impression I had was that we were leaving the \nWest and entering the East; the most western of splendid \nbridges over the Danube, which is here of noble width \nand depth, took us among the traditions of Turkish rule. \n",
        :page => 1
    },

    {
        :text => "We left in pretty good time, and came after nightfall to \nKlausenburgh. Here I stopped for the night at the Hotel \nRoyale. I had for dinner, or rather supper, a chicken done \nup some way with red pepper, which was very good but \nthirsty. (Mem. get recipe for Mina.) I asked the waiter, \nand he said it was called ‘paprika hendl,’ and that, as it was \na national dish, I should be able to get it anywhere along \nthe Carpathians. ",
        :page => 1
    },
]

And there you have it. Thanks to Tika, we easily split the PDF into paragraphs, and thanks to Nokogiri we have parsed it with extreme ease.

Getting the PDF Contents Into Algolia

So I decided to use Algolia after playing around with it, because it was extremely easy to use, configure, and it returned search results extremely fast. Getting my PDF content searchable was as easy as doing this:

Algolia.init(application_id: 'xxxx', api_key: 'xxxx')
index = Algolia::Index.new("books")
index.add_objects(paragraphs)
index.set_settings({ "searchableAttributes" => ["text"] })

Closing Notes

Now obviously you could add more to that example (maybe get the filename and PDF author/title in as a record too, or add book IDs linked to your system or whatever) but that’s a nice and simple example to get started from. I’d like to thank all the people who worked on any of the tools I’ve used because they’re awesome and incredible time savers.

Here’s the full code of the example as a GitHub Gist:

The Case of The Jumping Dot

Omar Bahareth — Tue, 19 Dec 2017 00:00:00 +0000

When you enter the dot/full stop character when typing in English, it just works, and you don’t need to think about it dancing around your words and sentences. This isn’t the case in Arabic. Since the full stop character is technically considered a BiDi (bidirectional) character, weird stuff starts to happen. There’s no clear standard on how to render it correctly(as far as I know).

Take a look at this GIF for example:

The dot suddenly went to the beginning of the line (on the right side instead of the left), and the blinking cursor isn’t even moving. Now this doesn’t just happen with that character, it happens with any BiDi character (e.g. exclamation mark) in the middle of RTL (Right To Left) text.

There’s actually a solution to this for some characters, there’s an Arabic full stop character, the only issue is that most people have no idea it exists or how to type it. I usually google it and copy it off of websites that show you Unicode characters, and it works pretty well as you can see in this example:

The way it’s rendered is a bit weird, and some fonts render it at a very strange vertical position, but it doesn’t jump around text and drive me crazy.

Another solution is to “set the text direction” but most users don’t know how to do that, it doesn’t work well when you mix LTR (Left To Right) and RTL in the same line, and it isn’t even immediately obvious how to do that in a number of platforms/apps. So I propose changing the normal full stop character to the Arabic one when using Arabic input (and the same for all other applicable characters), on all desktop and mobile operating systems. Which by the way, is what already happens when you try to use type a question mark with Arabic input.

I’ve actually tweeted to Apple, Google, and Microsoft, as well as their CEOs with no response (yet).

Could we get the Arabic full stop character to be inputted when we try to input a full stop with Arabic input? The normal one gets rendered all over the place and most people don’t know the Arabic one exists or how to type it.@Apple @Google @Microsoft https://t.co/ccpePNJ45G

— Omar Bahareth | Blockchain (@o_bahareth) December 8, 2017

@sundarpichai @tim_cook @satyanadella Could you guys get this looked at?

— Omar Bahareth | Blockchain (@o_bahareth) December 8, 2017

I would think these kind of UX issues are more important than where the cheese is on the burger emoji.

I think we need to have a discussion about how Google's burger emoji is placing the cheese underneath the burger, while Apple puts it on top pic.twitter.com/PgXmCkY3Yc

— Thomas Baekdal (@baekdal) October 28, 2017

Will drop everything else we are doing and address on Monday:) if folks can agree on the correct way to do this! https://t.co/dXRuZnX1Ag

— Sundar Pichai (@sundarpichai) October 29, 2017

There are many other difficulties that we face which I might show in other posts but I’ll mention just a few which I regularly face and am extremely irritated by:

Selecting Arabic text, particularly when there’s English text within the selection. What’s rendered on-screen as selected isn’t actually what’s selected.
Tiny font sizes. Most websites I visit have a font size of around 16px (Google search bar for example), this looks fine for English, but many Arabic characters are not legible for some reason. I don’t know if this is the fault of the font or if we need dynamic font sizes per language.

It is my hope that this post will bring some of the issues many users face when typing Arabic on computers and phones to light, and even drive developers and font creators to tackle these issues. There are many apps and websites that have solved a lot of these issues, but we need this knowledge to be shared and turned into standards that operating systems and browsers handle, without every app and website having to repeat the work.