DEV Community: shakycode

Blake the Onshore/Offshore Programmer

shakycode — Wed, 22 Feb 2017 00:33:53 +0000

If you've read my previous post The Story of the Fraudulent Coder you know that I have little to no tolerance for bullshit. This is another developer war story that I'd like to share with you. For the sake of anonymity I will be calling this programmer, Blake. Everything else in the story is a true tale of major pain and conflict, ok that sounds dramatic, but just bare with me.

So I was working for an awesome startup and we had a “Senior” programmer by the name of, “Blake”. Blake was a very polite guy who just happened to be on a H1-B Visa from overseas (India). We hired him with the intent to bring a stronger level of technical acumen onto our team since he had nearly 10 years of software development experience and 8 years of Ruby and Rails. We thought he would be an amazing addition to our team which really needed the help.

At the time our company was a 100% distributed team which meant we worked from everywhere and anywhere just as long as the work got done. We kept in communication via Squiggle to do video conferencing and chat and all was well in the beginning.

Blake was very quiet and was really hard to connect with as he always seemed to have something going on in his personal life. We initially dismissed this as being normal human nature as hey we all have shit going on but it's all about how you manage it. But after working alongside Blake for a month I started to notice some patterns and red flags.

Red Flag #1 (Missing in Action)

We kept lenient working hours but the majority of us were required to be online from 10:00 to 17:00 central time. Normally we would have our daily standup to start our sprint but most of the time Blake would have connectivity issues and not join our video conferences. He claimed to be on Google Fiber, but he was supposedly on the East Coast at the time and Google does not offer Fiber in his area. We would watch him bounce on and offline all day long. Part of working in a distributed team is having solid network access so that we can keep working and collaborate. We would see Blake online via Squiggle but whenever we reached out to him we'd either get no response or a canned messaged such as, “I'm analyzing this problem, I will report my findings later tonight”. The funny part was, there were no problems to be analyzed, he was doing basic programming tasks and simple debug triage which should take 1–2 hours max even for a newcomer much less a senior developer.

Red Flag #2 (Gender Changing Baby)

Blake was a father to a newborn daughter so often had to step away and tend to his child while his wife was at work. This was not a problem for us as many of us multi-tasked while getting work done. In talks with Blake we asked him if he had a big family and he said, “nope just my wife and daughter”.

A month later Blake didn't show up for work and we got a frantic email stating that his baby son had to go to the hospital because he was running a high fever so he would not be able to work today. My boss didn't catch it but I noticed that somehow his newborn daughter changed genders. Throughout our time with Blake he would switch between referring to his “son” and “daughter” when in fact he told us he only had one child which was a girl. Really whacky shit, but it gets worse.

Red Flag #3 (No Daytime Work)

We had a policy in place that we would be available during normal business hours to support our customers but we were flexible enough to where if you needed to duck out or take a break you could. We never encouraged our employees to work after-hours but many of us did because we were passionate about the work and vision. But we began to see a pattern emerge with Blake.

During daytime hours (roughly 7–8 hours) we would not hear from Blake whatsoever. We'd ping him on Squiggle or Slack and he'd not respond. Hours later in the afternoon we'd get the typical response of “I'm analyzing the problem and will have answers tonight”.

During the 3 months that we worked with Blake we never saw one commit during daytime hours. All commits were between 21:00–22:00 while the rest of us were steadily committing code during daytime hours and sometimes in the evenings. It's as if this guy was nocturnal. It ended up becoming a major problem because he was never available when the rest of the team was online and mid-sprint.
This was really suspicious and as you will read later on we found out why this was.

Red Flag #4 (My Wife's Car Broke Down)

We were in the middle of an extremely tight deadline and we were counting on Blake to help us ship this feature. Myself and 3 other developers were handling most of the work with Blake running point on the feature sprint. He supposedly had some code to commit and we were running out of time to ship. We put the pressure on him the day of the deadline and he emailed us, “Guys, my wife's oil cap came off of her car on the freeway and she is stranded so I can't work today as I have to get the car towed and get her to work”.

Um hello. Ever heard of AAA? Hell, 99% of auto insurance companies come with some form of roadside assistance. I mean it would be different if his wife was in a horrific accident and he needed to be here. But seriously her car broke down and she could have easily called a tow truck. Instead Blake decided that he needed to take care of it.

We all felt this was a bullshit excuse and we simply took over his portion of the work and shipped without him meanwhile almost meeting our deadline. But really this was not a feasible excuse and sounded like some sort of bullshit made-up story. We were all suspicious of him from that point forward.

Red Flag #5 (NodeJS/Cross Site Scripting)

We were working on a new API that would help us better aggregate and normalize data. Our app was monolithic and was installed on several different servers (one per customer) so we had to introduce this API into the codebase and pass parameters to it so that it would transmit data from the proper organization that the app instance belonged to (i.e. Acme Company).
During the development of the API Blake said that our API was subject to Cross Site Scripting and that we absolutely must use NodeJS to prevent the XSS attacks from happening.

Now I'm not a NodeJS developer but I know enough to know that introducing a full blown MVC JavaScript framework would not prevent XSS attacks and that this was something that we would have to handle internally via Rails. Come to find out there was no attack vector surfaced that would result in a XSS attack. We found this very odd and he kept pushing for NodeJS. Not wanting to bloat our tech stack and having zero need for Node we pushed back and said NO to Node, handle it in Rails. His response was, “I cannot be held responsible for the open vulnerability in our application and API”.

The funny thing is we sourced a very reputable security company to attack our staging server which ran the API and Rails framework and they were not able to execute a XSS attack or any successful attack besides a DDoS.

So where he came up with the notion that NodeJS would solve a XSS attack vector I have no clue. We brought this up to our CTO and he was not happy because progress was not being made and said, “Fuck Node, we're sticking with Rails and the API we are building”. Still weird as shit.

Red Flag #6 (Race Condition)

I was tackling a scaling problem at the time which was basically a SQL stored procedure that computed a scoring-based algorithm in realtime. This was happening during a page load and on clients with many users page load times would be in excess of 1–2 minutes which is entirely unacceptable. So I decided that we didn't really need realtime score calculations on every single subject in the database and decided to move it into a Sidekiq Worker (Background job) that would fire every 3 minutes outside of the request which ended up resulting in > 400ms response times on page loads and “good enough” realtime scoring data.

When I assigned the PR to Blake he rejected it and said that I'm introducing a race condition by not calculating in realtime. He went on to say that we would experience database deadlocks if we moved the scoring work to a background worker.

Knowing what I know about our codebase and race conditions this made zero sense whatsoever. So myself and another developer jumped his head and went to our CTO and told him about the situation. Blake was put on the spot by the entire team asking for an explanation of his race condition and deadlock theory but all he could come up with was, “I'm still analyzing the problem but will have a solution by tonight”.
If it smells like bullshit, it probably is.

Red Flag #7 (The 1 month Flu)

Blake had many excuses for not showing up to work or shipping code when we were trying to meet deadlines, however one in particular struck me as odd. One day Blake emailed us and said he had the flu and would not be able to work today. We wished him well and didn't hear from him for a week. We finally reached out again via email as he would not answer his phone and he responded that he was still sick and could not work. The funny thing was his wife was supposedly an ER doctor so she could have easily prescribed him some Tamiflu or other antiviral early on to combat the flu. I even asked him if his wife could prescribe him something and he said no because she was at work and couldn't be reached. This was very odd to me that he couldn't text his wife and have her write our a prescription for medication or have one of her physician colleagues do so. Or better yet, Blake could have simply sucked it up and driven to an urgent care clinic for treatment.

A few weeks passed and we finally got Blake on Squiggle for a video chat and he looked healthy as can be but kept a box of kleenex nearby even though his nose wasn't running. He would cough a lot but you could tell they were forced coughs and not a true symptom of illness. Now I'm not a doctor by any means but I know that a typical strain of influenza treated at home usually lasts 14 days max before it peters out. After 4 weeks Blake was magically “recovered” and asked what needed to be done. Sorry dude, we've been cranking out code for a month without you. Meanwhile you've been paid (extremely well) and have contributed zero to the team.

Red Flag #7 (I Can't Pair With You)

Being a distributed team meant we were heavy on pair programming. In my 3 months of working with Blake I paired with him once and it was really him just looking at code and then cutting me off and saying, “I need to analyze the problem, I will get back to you”. Now we're talking about small bugs like NILClass issues or data that wasn't normalized or simple 4 line functions that just needed a 5 minute refactoring to pass our test suite. Nothing major.

He would only pair using Teamviewer which is available on all platforms but we were an Apple shop and we gave brand new Macbooks to every employee including Blake so that we could use pairing tools such as ScreenHero to keep the productivity going. Blake refused to use the Macbook and always came up with an excuse that it was crashing on him and it didn't work. We even replaced his Macbook twice and he still complained that it didn't work. So the end result was we never got to pair with Blake and that was really the antithesis of our culture. It should be noted that Blake had an Ubuntu and Windows desktop that he would work on, both of these were NOT approved by our CTO due to security restrictions and the nature of our data. We had 2FA VPN connectivity and it was our company policy to only use our work laptops for work, not personal machines. Blake was breaking the rules from the day we hired him by refusing to use the company supplied Macbook we sent him.

So we tried our best to pair with Blake but each time we asked to pair up he would come up with the typical response of, “I'm analyzing the problem, I'll get back to you”. Then magically at 21:00–22:00 we'd see some sort of Github activity which relates to no daytime work as previously mentioned. And for the records the commits were horrible especially for a senior developer with this much experience. These were literally cut/pasted solutions from Stack Overflow with no credit given to the original source.

What We Found Out

Because of all of these excuses and bullshit factors we decided to do some investigation covertly and here's what we found out before we fired him.

Blake had multiple logins on his Ubuntu machine from IP addresses originating from APNIC (Asia Pacific Network) specifically India. There were multiple user accounts on his machine all of which had a fork of our github repo in their home directories. Yes, Blake was using offshore developers to do his work while he worked his day job.
We further found that he was offshoring his own job due to the time of his commits. In the 3 months that we worked with Blake we found his commits to be always between 21:00–22:00, no other commits were ever made outside of that time frame. Guess what? That's 08:00–09:00 in India.
We also discovered that he was working another full-time job with his previous employer on contract. How did we find this out? He made the mistake of leaving his old employer on his Linkedin profile and in a covert mission one of our developers called the company and asked to speak to him, the company said he was out on lunch but would be back in an hour.
We also discovered via github that his account was being used by another developer (or perhaps several). How did we know? Because their git name configuration listed another name and when we looked it up it was an oDesk developer from Bangalore. Busted!
We looked at Blake's github to see what sort of projects he was working on and there were a bunch of basic Ruby and Rails tutorial repos forked into his account. Was he starting an education service? Negative Ghostrider, the guy really had no experience.
We tallied up his commits for the 3 months that he worked with us and the end result was 8 commits and 27 lines of code committed. Meanwhile the rest of us were on long streaks and committing much more often. Blake was simply not getting the job done and when he was it was not his code. How did we know? Because we looked at the styling and comments and compared it to the oDesk developer we previously discovered and it matched to a T.
When Blake finally returned his Macbook after several threats of legal action for not returning company property it arrived in its original packaging that we sent out and was not setup whatsoever. He never used the Macbook come to find out and ended up using his Ubuntu desktop which was against policy.
After Blake was terminated one of us started scoping out his github and sure enough he had repos related to his old company which he never stopped working for. Blake had pulled the wool over our eyes for sure.

Lessons Learned

We all operated on the honor system and believe in trust and integrity. This was the foundation of our company. Unfortunately Blake had no integrity and could not be trusted. It's just a shame that it took us 3 months to figure this all out, meanwhile we turned down several heavy hitting senior developers because the position had been filled by Blake.

We learned a tough lesson and after Blake was terminated we upped our game as far as candidate vetting and ended up hiring legit developers who really took us to the next level.

My advice for those of you out there working with the Blakes of the world. Call them out on their bullshit early and fire fast. The integrity of your company and team depend on having legitimate, honest, skilled people. You deserve better, I know we sure did.

Such is the end of another dev war story.
shakycode â¤

Twitter: shakycode

Originally posted at shakycode.com

Rails ActionCable The Saga Continues

shakycode — Fri, 03 Feb 2017 17:25:52 +0000

Yesterday I wrote a post titled, “My Struggle With Rails ActionCable” which got a lot of good feedback and reviews. But overnight I thought long and hard about this problem that I faced in that I could not get the request origin aka TENANT to be delegated to the channel.rb file.

There was an open issue on the Apartment Gem where several people were discussing Apartment and Actioncable. After doing my own research and coming up short I saw a light at the end of the tunnel. One of the contributors to the Apartment gem found some documentation on how the user is verified for connection and he discovered the following in the documentation.

"Note that anything marked as an identifier will automatically create a delegate by the same name on any channel instances created off the connection."

So I went to work to refactor my code and this is what I came up with in the connection.rb file

If you notice the ActionCable connection is now identified by current_user and by tenant by setting the tenant in the connect method by simply calling self.tenant = request.subdomain then immediately following we switch the tenant and connect to the proper channel based on tenant. Since we are now delegating tenant to the channel it's a relatively easy fix to get the channel wired up and switch the tenant in our send_message method.

Now if you look at the send_message method in the channel file you'll notice we're switching the tenant as such Apartment::Tenant.switch!(tenant) which is possible now because we delegated tenant from the connection.rb file so that it's available to the channel.

End result, ActionCable and Multi-tenancy now working across the board. I honestly haven't seen a use case (trust me I've googled for hours) where someone was building a multi-tenant SaaS platform and integrating ActionCable.

So now this feature is near completion, just ironing out some coffeescript bugs and issues with the Background Job processor.

I'm hoping to write a full tutorial soon on this topic, but for now the problem is solved and I have to give a big thanks to the Apartment and Rails community for helping to steer me in the right direction to solve my specific use case.

What are you building with ActionCable? Are you implementing multi-tenancy? If so, give me a shout and let's compare notes!

Cheers!
shakycode â¤
Twitter: shakycode
Email: shakycode@gmail.com

Originally posted here

My Struggle With Rails ActionCable

shakycode — Thu, 02 Feb 2017 21:43:56 +0000

Let me give you a bit of background on this article. I'm building a multi-tenant SaaS platform that connects Physicians with Patients. Included in this is the ability to have realtime chat sessions between two parties. This is all built on Ruby on Rails, my framework of choice using Ruby which is my primary language. In Rails 5 ActionCable was introduced which facilitates dead-simple websocket functionality replacing the need for external dependencies such as FAYE or 3rd-party services like pusher.io. This is baked in and a pleasure to work with. Or so I thought.

You see multi-tenancy is a difficult problem to solve and ActionCable likes to transmit its websockets on a specific request origin. The request origin could be for example: https://acme.engagemd.co so all socket requests connect and stream from that request origin. If you aren't sure what a request origin is in Rails, think of a the request as where the HTTP request is coming from. In my case this is a multi-tenant app so the requests would come from a plethora of subdomain names. Actioncable looks for specific request origins based off of a whitelist that you specify in the production.rb environment file. Now you can get clever and use some Regex matching to allow subdomains from a parent domain i.e. acme.engagemd.co, doctor.engagemd.co, etc. But this is not enough.

Problem #1

In my new platform I am using the Apartment gem to give each tenant their own database schema and have written the platform so that the entire app will switch Tenants based on the request origin. When I first started writing a simple chat feature into the app I was presented with the issue of how to get ActionCable to connect based on the request origin. After some help from a friend and doing my own research I was able to get the connection.rb file which drives ActionCable to read the request origin and switch the tenant in realtime. This got rid of the rejected connections and the streams started flowing properly. This is my connection.rb file:

As you can see when we go to connect we set the tenant to the request subdomain then call switch on the second line of the connect method to switch the tenant and following that verifying that the user is part of the tenant and allowed to receive websocket traffic. That part was solved relatively easily.

Problem #2 aka The Show Stopper

In the channel file which is responsible for sending data to the client, there's really no way to access the request origin and switch the tenant.

As you can see in both the subscribed and send_message methods in the channel file it's required that we switch tenants. In the subscribed method I hardcoded a tenant just to see if it would switch and it did. In the send_message method (and the connect method) I really need to be able to access the request's subdomain so that I can switch tenants and livestream the chat.

Now here is the showstopper. After reading the ActionCable docs and Googling for what must have been 4 hours yesterday there's really no way to get the request into the channel file. That being said this is a complete failure and I'm not able to use ActionCable for multi-tenant realtime chat since the request is unavailable in the channel file.

So I went on a witchhunt trying to figure this god damned thing out. My first thought was to set a global like $request in the application controller and allowing the channel file to access that. This works in theory but has multiple problems.

Globals are a security risk and are prone to mutex conditions
When you have more than one tenant and user the global variable $request gets overwritten and ActionCable will not transmit after the tenant is switched. Or worse yet the global variable gets lost entirely and the tenant will attempt to switch but having no subdomain which to switch on fails and messages are not created.

I've opened up an issue on Rails core to see if anyone can help as the documentation is lacking and I'm pretty sure I can't be the only one with this specific use case. So I'm hoping for some feedback on this soon. In the meantime I'm tearing apart ActionCable in rails to see if there's a way to pass the request to the channel file. If I can figure this out I will do a PR against master and hope for inclusion in the next release of Rails, but I'm not holding my breath as the Rails community is an opinionated bunch and multi-tenancy is not what everyone is doing.

The Case for Polling

Considering the fact that I'm falling face first on my desk dealing with this ActionCable issue I've decided that a good alternative is do Ajax polling in the chatrooms where an ajax call is made every 2–3s and polls for new content and refreshes the div/partial. Surprisingly enough this is how Basecamp does it in their campfire chat instead of using ActionCable. I watched a good talk with DHH about Rails performance and he went into a bit how they are using Ajax polling for their campfire chat instead of livestreaming via websockets. Their thought was not realtime chat, but “realtime enough”. 99% of end users will not notice a 2–3s delay in chat which is plenty of time for the system to poll and refresh the partial showing new messages.

This is not what I wanted to build nor the way I wanted to build it but at this point I see no other choice but to move forward with Ajax polling and eventually ActionCable will introduce the request origin being passed from connection.rb to the channel file. Or I'll write a PR that introduces this behavior and hope that it's accepted.

Summary

Writing software is hard. Just when you think you have it figured out an 800lb gorilla hops on your back and takes you for a pony ride. I know that I'm making good progress on my platform and will continue to do so but all of these roadblocks can be discouraging. I'm doing my best to take these challenges as opportunities to grow and think outside of the box, but it's a bummer none the less.

Are you working with ActionCable and multitenancy? If so, let's chat and maybe we can come up with a solution that works for all!

Cheers!
Shakycode â¤

Twitter: shakycode
Email: shakycode@gmail.com

Hi, I'm shakycode

shakycode — Thu, 02 Feb 2017 21:33:14 +0000

I have been coding for 5 years.

You can find me on GitHub as shakycode

I live in Houston.

I work for many companies

I mostly program in these languages: Ruby, Python, Javascript, and Rust.

I am currently learning more about Event Driven Architecture and SOA Architecture.

Nice to meet you.