DEV Community: Jake Berkowsky

Managing a team's stress

Jake Berkowsky — Sat, 14 May 2022 22:46:49 +0000

As a leader, it can be tempting to push your team to get the most out of them and to get results. People don't like like saying no and often just want to be helpful and give it their all. Many people will even offer to do things like take a meeting on a personal day or work a bit later for a week or two. While it's important to respect people's privacy and their right to manage their own lives, it's also your responsibility to do what you can to discourage burnout.

I’ve found the best ways to manage this is to limit the amount of unnecessary stress at work, keep your ear very close to the ground, and use smoke signals. Some strategies

Lead by example, I once saw a manager leave a note in slack saying they were going to take that day off because they "woke up grumpy". That's awesome leadership, it encourages others to do the same and normalizes the idea of a mental health day.
Experiment with mandatory time off; my last role had flexible PTO so it was originally a safeguard to prevent people from not taking it. Apart from a minimum, we said that part of it must be consecutive. I made it clear that I would deduct from bonuses if people don't take enough.
Be clear and transparent. Don't make people stress out about unknowns. While constant reprioritization is a stressor, once you know about something it doesn't hurt to fill the team in. If you are setting expectations define them as much as possible and make yourself around to answer questions if/when there ends up still being unclarity.
Don't write or respond to emails after work hours; I may be working but I don't want to publicize (and certainly not glorify) it or compel people to respond. Gmail has a feature to schedule emails for the next day, so does slack.
Respect people’s privacy. Let people know they don't need to give a reason for taking time off and make sure not to ask for details unless people volunteer them.
Be there if someone is working off-hours. In my line of work, it was very rare for anyone to have to work nights or weekends. For the few times we needed to I made sure to be there (in person or virtually) as well.
Smoke signals; besides traditional 1-1s and retros, I did a weekly session where I take a bunch of gifs and put them on a miro board, then everyone goes around and talks about which one they feel most like. I had a "dumb manager" idea and created a dashboard to emulate a Brazilian steakhouse, (put up a green card if you need more stuff to do, put on a red if you are full), no one really used it but I don't regret trying.

At the end of the day, people will live their own lives. Stress can come from many places and while you can never eliminate it completely, it doesn't hurt to try to reduce it if you can. A less stressful team is more effective, happier, less likely to leave and just results in a more fun environment.

A response to “Thinking About the Future of InfoSec”

Jake Berkowsky — Mon, 02 May 2022 15:45:13 +0000

I recently read an article by Daniel Miessler that's been making the rounds about his predictions for the future of cybersecurity. He talks about changes to organizations, the market and the day to day of cybersecurity professionals. His post is definitely worth a read. I also decided to write down some of my thoughts in response.

The original article can be found here: https://danielmiessler.com/blog/thinking-about-the-future-of-infosec-v2022/

Organizational Changes

What will tomorrow's trendy org structure be?

Miessler's main prediction in this area is that security will transition from a discrete area to an embedded one. That it will fold into general engineering and operations and that it will be "[become] a smaller oversight function up with the C-Suite, with strong collaboration with the CFO and the head of legal."

Not going to lie, I like both of these concepts. Security is by its nature an exercise in risk management and most professionals will tell you that the human and policy aspect far outweighs the IT one. Organizations, where security falls under legal, are not unheard of and I wouldn't be surprised if it became more popular. However, I don't particularly see this as the "future" as opposed to an option that will be more appealing to certain companies.

Replacing a dedicated security organization with functions embedded into the rest of the organization is nothing new. Organizations, especially product ones, frequently shift from vertical to horizontal to a matrix and then back again to vertical. I feel this is more of a trend cycle than a one-way maturity function.

Day to Day Changes

These predictions continue to elaborate on the absorption of information security into the greater risk management effort as well as talk about the continued commoditization of the security profession.

A "Linux Admin"

Miessler describes a transition from "Wizards to Accountants" with the future state of security resembling a factory. I would agree that process repeatability is a sign of a mature organization and industry. However, I'd argue that the transition here has already occurred and is more dependent on the maturity of the company than the industry. The primary types of jobs Miessler predicts are connecting things together, doing data analytics and process management. Again these are the same types of roles I see in the enterprise today. When I work with a company, I choose a mature tool, an accepted framework and established processes for implementation. There's not a lot of wizardry that you wouldn't find in any other type of IT or infrastructure role. Additionally in non-security operations, which is a much more mature field, we still have plenty of wizards; there is just a lot of process and metrics built around it. A lot can be solved by good playbooks and automation, but there is still a never ending stream of obscure bugs that require a creative "wizard" type.

Miessler also talks about AI making a significant effect on the industry and that AI and automation will become inseparable. I think this is to some degree true. The amount of data generated by an organization has long surpassed the amount that can be manually reviewed. Automation is already needed and already exists. AI solutions, in my opinion, have not given us necessarily better results than traditional algorithms. They do however very easily improve the speed in which those algorithms can be developed. That being said, I hesitate to go so far as to say that they will replace analysts to any significant degree. I could see them just as easily generating more insights and thus increasing the number of analysts needed by an organization.

Changes to the market

The last type of changes Miessler discusses are those to the market itself. The predictions that stood out to me are that regulation will continue to have an increased role in driving requirements, that insurance companies will have more influence and become larger players, that the industry will favor general technology companies over security-specific companies and that the large centralized solutions will dominate the enterprise space.

I happen to agree with the first two predictions. Cyber regulations are becoming more commonplace and given the speed of government, it only makes sense that they will continue to mature over the next few decades. Cyber insurance is already a reason for many requirements and that they would become more involved.

The favoring of general tech companies over security companies seems reasonable. Already you can buy security products and even analysts from the likes of IBM, Microsoft and Google. However once again I wonder if Miessler is confusing maturity with trends. Conglomerates grow, then dissolve on a cyclical basis. GE was a huge company but now it's split, conversely, Disney is larger than ever.

One Ring to find them, One Ring to bring them into compliance and in the darkness bind companies to long term service contracts

The large centralized solutions that Miessler talks about remind me of SAP. While I don't strongly disagree that this could happen, I'm not entirely convinced it will. SAP's creation of an essentially closed garden and ubiquitous ecosystem was not the result of it beating out competitors but instead the result of it being the only alternative to creating it yourself. Today's cyber ecosystem is already full of vendor offerings that all integrate and connect to each other because they must. I can't imagine it going backwards.

Closing thoughts

I imagine that many of Miessler's predictions will come to pass. The question I see is not when, but for how long. A lot of these predictions have already begun. I feel many of them represent a future trend more than an end state. Still, Miessler does present a well thought out and interesting view that is worth considering and thinking about. All in all, I enjoyed thinking about the future and look forward to seeing if he makes any updates as time goes on.

WordPress on AWS, begrudgingly

Jake Berkowsky — Fri, 29 Apr 2022 14:50:41 +0000

WordPress and AWS have a lot in common. They are both insanely popular, very accessible, and even launched within a year of each other. They are both very easy to get started with and even easier to get tangled up in complexity if you're not careful. What follows is an overview of what it takes to run WordPress on AWS if you so choose to.

General Advice

Don't do it

Solid Advice

A bit of background about myself. I have a lot of WordPress experience, I've created sites, written plugins, and developed themes. I've run WordPress on single servers and deployed massively autoscaling and highly available container based solutions. I've deployed sites and multi-sites for family members and large enterprises. I even architected the original infrastructure for WP-Tide

This is why my site is hosted on WP Engine.

The reason is that self-hosting just isn't worth the time and hassle. WordPress is so huge at this point that companies that can host it for you are mature, affordable, and have all the features you need. There are a few good reasons for hosting on AWS however.

You must -- Orders from above or something. It's not your decision but you still need to implement it. This is where I get my experience from.
Your scale is huge -- Maybe you work for a massive company and your site takes in billions of hits a day. Maybe you have an enterprise discount with Amazon that gives you dirt cheap EC2 instances. I'd still go with Pantheon but if you're at an enterprise and the one building it then reason 1 probably applies here as well.
You want to -- You really don't care about cost or headache because you're planning on hosting this on a server for fun, or maybe you want to learn AWS. Great!
You have some really custom stuff going on -- If you need direct control of the underlying VMs, if you have modified the core, or if this needs to be on your network then your only option may be to self-host.
It's dev -- Ok fair enough, if you're a WordPress developer and need to constantly spin up dev instances for a few min then yeah go and self-host.
You've already been at it for a while and it would look bad if you gave up now.

You'll notice that saving money isn't on here. Once you add in things like VPCs, managed NATs, Load balancers, patch management, backups, and employees' time self-hosting rarely saves you money.

If you're going to do it anyway, keep it as simple as you can

If you're going to self-host on AWS try to at least be boring about it. I've seen people (including myself) get very excited about best practices. It feels good to build something right. To build it scalable, to automate everything. However... In this case, I'd recommend doing it simply. Don't think about a massively scalable docker based system or building the entire thing to self bootstrap (I've done both). What you should be focused on is getting a single server that patches and is easy to restore if something happens. If you need to scale, you just make it bigger.

Comparing Approaches

Single Web Server

If at all possible try to keep it to a single web server. You can always decouple the database for additional scaling and control but remember that WordPress wasn't designed to run on multiple web servers. AWS offers very large compute instances so in all likelihood a single server will almost always give you enough power for your site.

WordPress can be deployed right from the Lightsail console

Lightsail

Lightsail is Amazon's version of Digital Ocean. It's comparable in cost and has the option to convert whatever you transition to "real" AWS when you're ready. If the goal is to "self-host on AWS" congratulations this is it. You can get up and running in just a few minutes and because it's Lightsail you don't need to worry about hidden charges. Once it's up and running make sure you enable unattended-upgrades for OS level patching (you'll still need to update WordPress separately) and enable snapshots. The best way to get started is by using the wizard, you can find a video guide and resources here:\
https://aws.amazon.com/lightsail/projects/wordpress/

EC2

If you have special networking needs, or already know AWS pretty well and want it integrated into your environment then you can just build it in EC2. You can either use an AMI from the marketplace to get started or just install it yourself. If going this route and you want to "scale" I'd recommend also setting up an RDS database. You can add a load balancer for SSL termination and have AWS automatically rotate the certificates for you and make sure you enable scheduled AMI and/or RDS snapshots for backups. For OS patching you can use either unattended-upgrades or AWS patch manager. A benefit of this method is that it can be pretty well automated with CloudFormation. Such as with this AWS Labs template:\
https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/solutions/WordPress_Single_Instance.yaml

Multiple Web Servers

You almost certainly don't need this and it will lead to problems both with the core application and with many plugins. WordPress was always designed to run on one server and backwards compatibility is a core principle of its development. Here are a few things to consider:

Complexity; instead of storing files and session information locally, you'll need to create a way to do both. This looks easy on a whiteboard but implementation is hard and requires changing a lot of things you didn't know existed.
Updates and upgrades will be harder, everything needs to write to the right place, refresh properly, and be coordinated. Upgrades will be harder. You"ll probably need a dedicated admin node.
Cost: this will add a lot more resources
Plugins and themes will break; oftentimes developers assume that their software runs on single server implementations. Though it's possible to make plugins that work fine otherwise many don't support it since the market is not large enough.
You'll be on your own. WordPress has a fantastic community with a lot of people sharing knowledge, the amount of people doing a multi-server setup on the other hand is very small and usually limited to larger companies that don't share externally as much.

If you want "multi-az" for reliability reasons then consider that this setup will almost certainly lead to more downtime due to errors and unexpected behavior.

If you're doing this for performance reasons just remember that a single web server can handle a lot of traffic if it's big enough. Unless you're serving billions of requests you probably don't need it. If your site is slow and you're reading this article because someone at Amazon told you this is the "best" solution. Take a step back and read this article first:\
https://wordpress.org/support/article/optimization/

Elastic Beanstalk

This is the old standard, Amazon has a bunch of labs and tutorials for how to do it. If you need to go multi-server this is a good place to start:\
https://aws.amazon.com/getting-started/hands-on/build-wordpress-website/

Architecture Diagram showing Elastic Beanstalk solution

Custom solution

This would probably make a good standalone article. However, if you're going to be building this yourself here are some pointers:

Use Amazon EFS for the WordPress directory. You'll get a bit of a performance hit but it will make upgrading easier and hopefully mitigate any plugin compatibility issues.
Create a dedicated Admin Node, this will be used for all uploads, database updates, and upgrades... it can be smaller than your other nodes.
User uploads (if needed) should go to S3, use a plugin to make this happen. Otherwise, the user experience should be read-only
You can set up AutoScalling on EC2, or use ECS. Although there's really no reason to containerize this. This isn't micro-services and you're not going to be constantly redeploying the application.
Terminate SSL at the load balancer. This will get WordPress confused on whether its getting encrypted traffic or not. Set the following in your wp-config.php file

if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https') {
    $_SERVER['HTTPS'] = 'on';
}

So there you have. In short, it's cheaper, easier, and more stable to pay someone else to host for you. If you still want to self-host do it on a single VM or with at most a separate database server. If you want to be more complicated than that review the tips and good luck!

Helping engineers make better estimates

Jake Berkowsky — Mon, 25 Apr 2022 14:46:26 +0000

One insight that I've found pretty ubiquitous in my career is that engineers hate being made to provide estimates on things. Still, in my role selling consulting engagements and projects, estimates were almost always needed and engineering input has been essential in making them. As an engineering leader, I've developed a few "best practices" that I'd like to share here:

Never blame engineers for discrepancies or "hold them to it" Estimates are by definition subjective. It's important that engineers feel comfortable making these estimates, don't start to see them as "deadlines" and definitely don't feel bad about not guessing the right number. It's their job to make their best guess with what they know and its your job turn those insights into a formal estimate, set benchmarks and timetables based on them and adjust and communicate with the client if they need to change. If/when the estimate is off resist any urge to blame the engineer, even in jest. Engineers should be focused on making an accurate estimate, not on setting the minimum reasonable goal. Things change, and if you don't embrace that engineers will either overly pad or feel an unnecessary stress to make these deadlines. For companies that bill hourly this is doubly dangerous. An engineer working "off the books" results in untracked and unbilled hours and means your future estimates will be just an inaccurate resulting in the same stress for the next engineer.
Track everything Keep track of your estimates, who guessed what, what the end result was, what if anything caused the deviance. This will give you valuable metrics, allow you to understand how well each engineer guesses (I would often have my own padding ratio per person) and when shared with the engineering can lead to them become better at estimating.
Create feedback loops and knowledge sharing Embrace the retro process, really dig into what the unknown unknowns were. This will allow the individual engineers and the organization itself to get better at making estimates. For new projects, you'll now have the option to consult both the engineers who will do the work, those done it in the past, and the retros if those engineers are no longer on your team.
Engineers should always get a chance to review the estimate before its submitted On my projects, I would always let the proposed engineer (if known) take a look at the final proposal before we sent it out. I would include the business and pricing information as well since I prefer to run a transparent organization but if this isn't possible at least make sure you share the timeline and hourly estimates. This gives the engineer ownership (although once again be careful of rule #1) and adds that extra set of eyes of things you may have missed (vacations, side effects....).

These are my personal rules, they come from years of creating estimates at a cloud consulting company and of course other types of companies and product may deviate. Got thoughts of your own? Let me know on this twitter thread.

Critiquing cloud lockin

Jake Berkowsky — Tue, 19 Apr 2022 13:09:05 +0000

I hear a lot of talk about cloud lockin. I hear it from people with self funded startups, authors on tech blogs and developers. The argument I hear most is that if you start using cloud native tooling, you’ll become dependent on it, accumulate tech debt and be forever burdened. There are of course legitimate reasons to be cloud agnostic, such as if your customers actually care what cloud your on, if your making a product that customers can self host, or if it makes compliance easier. In most other cases, I believe that a cloud agnostic approach incurs many of the same costs as a cloud native one, that the cost of migrating away from a cloud native solution is overstated and that the gains from using cloud native technologies outweigh any technical debt accumulated.

There is an idea that cloud native solutions impose technical debt, that the initial convince is offset by added complexity or cost later on. This may have some truth, but I’ve yet to see anything indicating that this is not also just as true for cloud agnostic technologies as well. There is a fear that your cloud service may cease to exist or cancel a service on you. This is an inherent risk of all Saas’s cloud or otherwise. The large cloud companies are generally financially stable and not going to go out of business anytime soon. Services may be deprecated but it depends on the cloud, AWS still supports services on its API that have been phased out 10 years ago. GCP is more heavy handed.

Another concern is that the price of your existing services will go up. This is possible but not for the obvious reasons. Generally speaking, prices for services go down as the clouds get bigger and rates get lower as you consume more. It may be that you reach a scale that the costs of managing your own workload outweigh the costs of using a managed service, but at that point you’ve already saved time and money to help offset the cost of refactoring. Moreso, the inherent scalability of managed services gives you time and breathing room to refactor at your own pace. I’ve heard people talk about hidden costs that don’t show up until a certain point before hitting you big. This is also partially true; some services charge per use or by bandwidth, this could mean that some inefficiencies won’t be discovered until you scale and can lead to unexpected charges. This is why its important to monitor and set billing alerts, especially after you deploy large changes.

If the cost of migrating away will be higher than the savings gained by leaving you are locked in. There’s little direct cost involved except for maybe bandwidth but that’s generally negligible; in fact if anything a competing cloud company (or one of their partners) may even incentivize you to move. It’s true that migrations can be a painful and expensive, but that’s generally not because of the cloud. More likely the culprits are things like uptime requirements, lack of documentation and the fact that no one has restarted the servers in a few years and don’t know what will happen when they do. Often times, its not much harder to migrate to another cloud than it would be to migrate to a new account in the same cloud.

There is the fact that your engineers will need to learn new technologies. This is a real cost, although it can be mitigated by a phased migration approach. Going mutlicloud can allow your engineers time to learn the new services before you completely move everything over. Some services API’s are also flexible. There are adapters that allow you to use the S3 interface to access files on other clouds blob storage solutions.

Cloud providers generally strive to provide service parity. Additionally, many of the services that cloud providers offer are in based off of commonly used or open source projects to begin with. This means that even if another cloud doesn’t offer a similar service, you can likely still use the original product it was based on. A multicloud solution may give you the best of both worlds, allowing your engineers more time to learn the new cloud and allowing you to pick and choose which services you want from each provider. Certain things can not be easily migrated. Things like the fundamental building blocks like security groups and IAM may have different paradigms that require a refactoring. Some formats like machine images can’t be natively exported but there are native tools that can not only move these formats but can move entire workloads. Many of them are provided free of charge by the competing cloud providers themselves.

Cloud lockin is an ideal that’s often overvalued. Leaving the option open for a migration could be a good idea, but cloud native services can provide an easy and scalable way to build your workload. Its important to understand the tradeoffs as you build. Acquiring technical dept and increasing your exit cost is not an inevitability. Research before building, adopting a multi-cloud approach and embracing refactoring can mitigate the cost of a future migration. There are valid reasons for going cloud agnostic but there’s no reason to consider it a “best practice”. In short, do whatever gets you in production in a quick, stable and secure way. Happy building!

Shifting left with vulnerability management

Jake Berkowsky — Sun, 17 Apr 2022 18:21:40 +0000

Recently a friend of mine told me his company, in an effort to improve security, was launching a bug bounty program. I’m a huge fan of bug bounty programs, hiring professionals to test your code is a great way to find things you may have missed and lets your clients, employees and investors know that you take security seriously. I asked him about his company’s reasons for doing to, he talked about protecting data and keeping their systems online but also mentioned that by outsourcing the bug finding they hope to save developers time and be more efficient. This shocked me, pentesting and bug bounties, while important, are the most expensive and inefficient part of the software development lifecycle.

The most expensive time to fix any software bug is after it has been in production. Refactoring code means you have to remember what you wrote, or decipher what someone else wrote. It means you have to be careful not to break anything. It means you need to push back improvements you’ve been planning and tech-debt you’ve been meaning to take care of. Paying per vulnerability is expensive too. Imagine paying a fee every time your linter discovered something, or every time a test fails. It’s inefficient, if the researchers don’t have access to your source code or your developer’s expertise, it may take them hours of research to find something that someone with full insight could have found much more quickly. It’s far better to tackle vulnerabilities with a devops mentality. To save time and money, you should shift left, discovering vulnerabilities as early as possible though scanning, linting and training in secure development at all levels.

If your application is public facing, congratulations, it is likely already being scanned. If your application or infrastructure runs on a popular platform such as wordpress (or windows) bots are actively looking for vulnerabilities to infect you. Sadly, unless something breaks you’re unlikely to know what’s being scanned for and what was found. Companies would be wise to scan their environments, both in production and non-production. Production scans should be less intrusive than their non-production counter parts. By scanning your pre-prod environments you’ll be able to catch certain vulnerabilities before you deploy.

Static and dynamic code analysis can find both obvious and obscure vulnerabilities at a fraction of the cost of employing people to do it manually. Like most software testing, the best time to run these tools is all the time. You can build linters in your CI/CD workflow and trigger them on commits, you can even have them run when a developer saves a file. One project I’m on has even installed IDE integrations to catch potentially vulnerable code before the developer even finishes typing it. Like the scanners, there are many tools that offer this functionality, ranging from open source to enterprise level.

Shifting further left, training and educating developers, architects and managers in secure development will allow for vulnerabilities to be squelched even before writing a single line of code. Employees can be educated on techniques and tools relevant to their position and security can be top of mind while architecture the application and even when writing the requirements. Catching and remediating vulnerabilities is best done as early in the development process as possible. Scanning your application regularly, testing your code and training engineers and management in security best practices will help make your code more secure while saving you time and money. Perhaps there will be a day when all security problems will be solved and we won’t have to worry about it. Till then, it’s best to scan early and scan often.

Calling the brute(force) squad

Jake Berkowsky — Thu, 14 Apr 2022 14:04:13 +0000

I got this picture in my family chat recently with a the question "is this correct?" The short answer is "kinda". The long answer is this blog post 🙂

What is Brute Forcing

Put simply, brute forcing means that a password is guessed. This is as opposed to it having be stolen, or intercepted circumvented via other means. Typically its guessed without the attacker knowing any other information about the target personally (other passwords, pets names...).

In order to calculate the time needed to brute force a password you need to know two things. How long does each guess take and how many guesses on average do they need in order to guess the password. Its important to know what is being guessed and how. Is this someone trying a login form on a website? Trying to unlock a phone they have in their possession? Maybe they're trying to decrypt a file. Typically though, charts like the one above refer to cracking a hash.

What's a hash?

A hash is the result of a "one way function" that takes in some data (call it "plaintext") and returns other data of a fixed length (a hash). "One way" means its not reversible. You cannot determine the plaintext from the hash.

For the sake of illustrating lets design a simple to understand hashing algorithm (method). It takes in a number, and returns the digital root, which is the sum of all the digits, taken until there is only 1 digit. For example, the digital root of 135 is 9 because 1+3+5 = 9. The digital root of 862=7 because 8+6+2 = 16 and 1+6 = 7. This is a hashing function because it takes in arbitrary data (any number), returns a fixed length hash (1 character long) and is irreversible (a hash of 4 can come from 31, 94, 22...)

In practice (for reasons we'll discuss later) hashing algorithms are more complex. Common examples include MD5 and SHA256.

There are a few reasons why people might want to use a hash instead of storing the plaintext directly. For one, it saves space, while the plain text may be of any length a hash is fixed and usually on a few character. For instance the MD5 hash of the small phrase "it was the best of times, it was the worst of times" is 4C0D0D10C00D6794492E86E3175F0A55, which is the same length of the hash of the the entirety of Shakespeare's Henry XIII 0449dd00c18ff30e70637dbc915912d4.

When storing passwords in a database this is more secure than storing the "plaintext" because if the database is breached, the attacker does not get the whole password, just the hash. At which point, if they want to recover the password, they need to "crack the hash".

Guessing speed

If the attacker is guessing passwords via a website login form they have to wait for the request to be sent to the remote server and returned. Perhaps the remote server limits how fast an attacker can guess passwords, in that case it may take a very long time to make a guess. This is what happened in 2014 when attackers broke into the iCloud accounts of various celebrities, they simply "guessed" the passwords. Since then iCloud has implemented a limit to how many times a user could enter a wrong password before locking the user out. On an iPhone, there is a physical backing off. The more times someone guessing wrong, the longer they have to wait to try a new password. In this case, brute forcing even a 4 character password could take decades.

Limiting the possible options

I once received an encrypted document via email. In the email, there were instructions which told me that the document itself did not contain any personal information (for my security), it went on however to tell me that the password was my social security number! By saying in the email that the password was a social, the attacker now knows they can easily brute force the encryption, because a US social security number is 9 digits.

9 digits is 1 billion options, which means on average a computer will guess the password after 500 million tries. For a computer this only takes a few seconds. For comparison, if the password could have upper and lowercase letters and number then each character would have 62 options, a 6 character password then would have ~56 billion options (62^6) and would take about 56 times as long to brute force. There are also 34 special characters, so a "secure" password of 8 uppercase, lowercase, numbers and special characters would have 6,634,204,312,890,625 options. Which is on average 3 million times harder to brute force than 9 digits. This is the basis of the chart above.

(Importantly, none of this matters if your password is p@ssword1. Hackers will likely be guessing common passwords before they try brute forcing. Password reuse attacks are also more common and easier to pull off than brute forcing. So easy that after the LinkedIn breach of 2016, other non-breached sites chose to reset **their* users passwords just to proactively stop it.*)

Collisions

If a hacker is brute forcing, they still don't need to try every possible combination. In fact, because of the way that hashes work, there are infinite amounts of plain texts that will result in the same hash. This is called a "collision". To use an the digital root hashing algorithm as an example, a user may set a 6 digit password of 938533, this is stored as "4" in the database. In this case, 6 digits is 1 million options (average 500K guesses). However an attacker doesn't need to try that many, they really just have to try 1-10 (average 5) guesses. Since 4,13,22,31, and 938533 all result in the same hash (4), this made up system will accept all of them as correct passwords. In practice since hashes are usually longer than your average password, collisions are generally not this easy to find. That is unless there is something wrong with the hashing algorithm...

An "ideal hash function" is a hash function that essentially acts as a random mapping between inputs and outputs. If a hash function was ideal, that means that the only way to find a collision is by brute forcing. In practice however, hash functions are not ideal and even the best ones we have are constantly being challenged. The MD5 hashing algorithm for instance was once considered secure, but was later discovered to have serious collision vulnerabilities making even the strongest password trivially crackable.

Hashes stored incorrectly can also causes problems. Since a plaintext will always result in the same hash, an attacker could come up with a list of mappings between plaintexts and hashes. Allowing them to save time, only having to calculate the hashes once. These exist and are called rainbow tables. Rainbow tables can be fairly large, a list of all hashes for 8 characters can be several hundred terabytes. Larger than what most PCs have available but can be stored on the cloud for a few hundred dollars. To counter this, applications can "salt" the hashes. Which means it adds some random characters to the plaintext before storing it. (the random characters are stored with the password to help with verifying passwords later). Those extra characters make rainbow tables infeasible (a rainbow table of 20 character passwords would take up more space than all the hard drives ever made).

Putting it all together

So what we've learned here. Brute forcing is guessing passwords, the more characters the more possibilities and the longer it takes to guess. Strong passwords with more types of characters are harder to guess. Still, that's not the only thing that matters. Weak hashing or storage choices can compromise even the most secure passwords, and a common password is not secure regardless of how many special characters you use.

If you really want to protect yourself, the best advice is to use a password manager (lastpass, keepass and 1password are all good solutions) to generate and store your passwords. Don't reuse credentials to prevent against a reuse attack and enable two factor authentication when available.

Security doesn’t have to be a blocker

Jake Berkowsky — Wed, 13 Apr 2022 14:51:34 +0000

A few months ago during a conversation at a secops event, the topic of granting exceptions came up. One of the attendees was shared his dismay. “Management is always steamrolling me” he complained, “people are just being lazy, they should be able to do it right” and added “if it were up to me, I wouldn’t give any exceptions!”

I can empathize; even given recent trends, security is still often thought of as a necessary evil, a blocker to get past as quickly as possible. This is no good. If you’re a blocker people will avoid dealing with you; you won’t find out about things until its too late to fix, be forced to accept the risk and be held accountable if that risk materializes.

I take the opposite approach, I grant exceptions freely and easily. I strive to present myself (or my team) as partners and aim to create a culture where we’re not seen as blockers. The goal is to bring value to the company by managing risk and more often than not that means accepting it. This requires a bit of bravery, you as a security professional must personally accept that risk. It means that yes, occasionally that risk will materialize and you will need to defend your decision. However, by presenting security as a partnership as opposed to a blocker, will will gain more visibility, you’ll be able to document and track threats, better prioritize remediation and get a chance to mitigate as well. In short, you’ll be more effective.

Imagine an engineer needs to expose a DB to the public internet so that an ETL job can run and executives can get their dashboards. You can say no or demand the engineer rearchitect. This may work, but now the dashboards will be delayed and it will be your fault. Or perhaps the engineer will escalate, you’ll get steamrolled and then just have to hope that nothing goes wrong. Instead if you work with the engineer you can mitigate. Allow a public facing DB, but ensure authentication is setup correctly and lock down the firewall so it only allows from the specific service the ETL needs. Next time the situation comes up, the engineer might approach you earlier and you’ll end up with a working (security approved) pattern that can be used again and again. This scenario may sound like its right out of an ivory tower, but it’s not. This exact scenario happens all the time.

Naturally there are always situations where exceptions cannot be granted. You can’t just break the law and there are many situations where you “can” be as much of a blocker as you’d like. Banks, government and medical device companies have a much higher tolerance for “dealing with” security. Still, even if you are unable to grant an exception, adapting a partnership mentality, helping to brainstorm solutions, encouraging consultation early on in the process and explaining your rational will make you more liked in your company and will make your job easier.

I have a colleague who was once approached hostilely by a high level and influential executive. The executive accused him of “blocking” the project and threatened to have him fired. My colleague turned this into a constructive conversation, explaining first that breaking the compliance regulations would have devastating effect on the entire company, resulting in the loss of not just this one project but others as well. He pointed to examples of how within those constraints, his decisions and policies were geared toward moving things forward. He showed his commitment to help achieve the goals of the project and the company and to adding value. At the end of the conversation he had gained an ally, the executive thanked him for putting the company first and sent an email to her staff showing her support, reminding them that security was not optional and that the staff would have to play ball.

This is a new world for security. We have the opportunity to provide real value for our companies and customers. Security, while once seen as a cost of doing business is now ammunition for sales and marketing teams. By presenting ourselves as partners we reduce risk, improve visibility and gain allies.

A Threat Overview of Contact Tracing technology

Jake Berkowsky — Tue, 12 Apr 2022 18:32:30 +0000

This past year, as the Covid-19 virus began to spread so did the efforts to digitize the contact tracing process. As fast as the virus grew, so did the number of technical efforts by countries, institutions, enterprises and hobbyists 1. While people across the world showed themselves eager play their part in helping to overcome the virus, many expressed concerns over the security of their personal health and location data. This article will attempt to give a scoped overview of the current Covid app ecosystem and explore in-depth some of their security implications.

An Overview of the Covid-Tracing Ecosystem

Covid tracing apps can largely be separated into two distinct camps. Those which attempt to associate an individual with a specific time and place, and those that focus on associating individuals with each other.

Time and place based apps

The efforts focused on associating individuals with a time and place, allow health agencies to better focus on retracing the steps of individuals, identifying potential hotspots and better informing individuals of potential exposures. They tend to work either individually (through contact tracers or application alerts) or in bulk (though publication of hotspots or paths taken by potentially positive testing individuals).

Instructional poster for SafeEntry

SafeEntry was one of the first technical efforts to receive mainstream press attention. Implemented by the Singaporean government, SafeEntry is a checkin based system. When members of the public enter a school, workspace, hotel or taxi they will see either a posted QR code or business maintained reader, and can either scan the code with their device or scan their device or national ID card. They may also checkin manually through an app and are encouraged to "check-out" when leaving one of these places. By law, certain types of businesses are mandated to participate in this program either actively or by posting a QR code in a easy to see place. Checkins are stored centrally on government servers for "as long as needed" and are used to assist health agencies.

CovidSafePaths, developed in coordination with MIT uses GPS data provided voluntarily by positive testing individuals. The app itself has 2 parts. In the first, a users location is tracked once every 5 minutes and saved locally to the users device. If the user tests positive they may voluntarily share this information with public health authorities. The second feature of this app allows public health authorities to alert users to potential exposures. In this scenario, users will download information about potential hotspots and compare them with the locally stored location data. A match will alert users to a potential exposure and possibly provide information for next steps.

The apps above rely on user submitted data sent to individual agencies. The data is fragmented, incomplete in what it collects and depends on large scale adoption. An alternative is to use existing data sources and collectors. A wealth of location information is already being collected by telecoms, corporations and governments through already installed apps, location service providers, Bluetooth ad beacons and various other means. In mid-march, the Israeli government announced that it would be using information acquired though its "Shin-Bet" internal security service to cross reference data collected through cell phones and "special means" for use in contact tracing. Google has released "aggregate, anonymized" community mobility reports to help health agencies determine if the effectiveness and need for social distancing policies. US based SafeGraph, a commercial provider of geospacial datasets, expanded its metrics to include information about social distancing, which could also help contact tracers in determining if an area is "high risk".

Connection based apps

Other protocols and apps focus on the connection between individual devices than on linking specific individuals with specific times and places. These apps rely largely on Bluetooth and involve devices sending messages to each other. This method generally aims to be more privacy-centric and is seen in some of the more popularly covered apps and frameworks. Once again, these efforts can be centralized or decentralized.

The Bluetrace protocol is currently used by the TraceTogether and CovidSafe apps, implemented by the Singaporean and Australian governments respectively. It aims to assist a "centralized contact tracing operation" with "decentralized recall". These apps continuously measure the distance and duration between two devices running bluetrace base apps while exchanging messages between the devices containing temporary keys. The protocol provides a means for users to share information with public health authorities and for health authorities to contact potentially infected individuals. When a user is confirmed positive, they may choose to share their encounters with public health authorities who in-turn may contact other potentially infected users. Bluetrace also supports a method of federation between health authorities using different apps while still attempting to respect the privacy of individuals.

While the Bluetrace protocol itself allows for a decentralized option without the need for centralized contact tracers, the authors of the protocol recommend against it. They opt instead to keep a "human in the loop". To quote the authors: "because a human contact tracer would seek to in-corporate information beyond just physical proximity,he/she can correct for systematic biases introduced by a purely automated notification system" 2 . They also note that contact tracing may produce anxiety which can be helped mitigated by person to person interactions.

Bluetrace promotes the idea of using a centralized backend to generate temporary keys, which may be downloaded in bulk to allow the app to continue functioning in areas without internet access. The authors acknowledge that it is possible to generate temporary keys on the device using assemetric encryption, but advise against it as it requires more computing and battery power and argue that centralization of key generation can serve as a secondary way to track app adoption. This functionality means that in order for this app to work, a user must register with the public health authority operating the app.

The Exposure Notification API, also known as the "Apple-Google" approach is also serving as a basis for many new apps. In a way, it is the "official" way to introduce contact tracing apps to the market due to the fact that Apple does not let Bluetooth tracking Apps run in the background if they are not using this API. The Exposure API itself is based on the DP-3T protocol which aims to allow contact tracing while preserving privacy. Like Bluetrace it relies on individual public health agencies for the actual app implementation and instead focuses on providing the protocol and framework for potentially federated contact tracing. Unlike Bluetrace, the Exposure Api is made to be decentralized with almost no identifying data collected by public health agencies by default (although it may include a mechanism for agencies to request information). With the Exposure API, temporary keys are generated on the device and never shared with a public health authority unless the user intentionally uploads them. The master key which these keys are derived from is ideally never shared at all. When a user is confirmed positive and uploads their data, their temporary keys are published publicly and other devices check to see if they have recently encountered those keys for a prolonged period of time.

How the Exposure Notification API works

False positives, spoofing and replay attacks

In various situations a user may end up reporting false positives. This can potentially be a user mistakenly or maliciously self confirming a positive test, a bad actor attempting to either mislead the system or impersonate a user or a carry out a replay attack. In a replay attack, previously sent messages of a positive testing user are resent in order to simulate a fake outbreak. All of these scenarios can have an adverse impact on individuals and the community as a whole as it could cause panic, distrust in the system or misallocation of a health agencies already limited resources.

Individuals wrongfully reported or alerted to exposure may face an unnessarary quarantine. In Israel, people have been ordered to quarentine based on cell phone Exposure data alone, such as a woman who was only waiving to her boyfriend from outside their house. In another incident, contacts of a health employee (including coworkers) were ordered to quarantine after mixed up lab results. Even after the error was discovered the employee was still unable to contact those affected to let them know it was safe to come back to work 3. Even if not under legal order, individuals receiving contact alerts may themselves impose a voluntary "self quarantine". No matter the cause, unnecessary quarantines can have detrimental effects on individuals, causing them to miss out on work related income, be unable to care for others such as aging relatives or patients and can prevent them from seeking unrelated but still potentially critical medical care themselves.

Uploading data with TraceTogether

Having a "human in the loop" is a strong mitigation against these types of attacks. A human can verify positive tests, sanity check situations like those described about, can verify potential "replays" and can give context to those who receive and alert saying that they've been "exposed". This is the preferred behavior of the Bluetrace based apps and is how Covidsafe and TraceTogether work. The Exposure API does allow (but not require) a human to verify positive tests and and choose which of a users temporary keys are published. However, Exposure API based apps require an affected user to proactively reach out to a contact tracer when presented with an "exposure alert". To prevent users from unknowingly sharing information with bad actors impersonating officials, Bluetrace provides verification in the form on PINs that contact tracers must provide when requesting data.

Technical measures can also be put inplace to prevent against replay and impersonation attacks. Encrypting metadata and including timestamps in shared messages can help limit the attack for Bluetooth based approaches. Using temporary keys can prevent individuals from "minting" new messages unless they otherwise compromise the "seed" those temporary keys are created from.

Unwanted location tracking

There are of course types of users who don't want their physical location tracked, either directly by the health agencies/app providers or indirectly by third parties exploiting the apps functionality.

Limiting the information collected and the amount of time its stored can help with this. TraceTogether removes locally stored information about encounters after 25 days. As mentioned above, CovidSafePaths only takes location information once every 5 minutes. This reduction in granularity reduces the amount of information that can be obtained on an individuals habits. Apps may also choose to fuzz the exact GPS coordinates of an individual. Both of these approaches will help mitigate location tracking concerns, though it is up to the health authorities to decide if this degrades the effectiveness of their apps. For CovidSafePaths, sending location data is "opt-in" and should only be submitted when a user tests positive for the virus. This greatly reduces the amount of data collected. SafeEntry collects and stores indefinitely all checkins by its users, however, this information is also in a way "opt-in" as its up to users to choose if and when they manually "check-in" at a place.

The Bluetooth model is attractive in this regard because it does not log location, but rather relationships with other peers. In the Exposure API model, the app is specifically forbidden from requesting location permissions. This is incompatible with certain older versions of android however which require the location permission in order to access Bluetooth. This is the case with TraceTogether, however they explicitly state that location is not collected. This and other threats are mitigated by the fact that the application is open source, allowing users to verify these claims themselves.

If an advisory knows the seed used to generate temporary tokens, that party may be able to use that information to track a user in real time. This assumes they have a means to listen for messages which is not beyond the realm of feasibility as many stores and locations already have Bluetooth trackers installed for use with advertising. In the Bluetrace model, the seed is stored centrally by the provider. This increases the likelihood that the central authority will be able to exploit it, as well as the impact of a breach. Storing the seed and generating the tokens locally reduces this risk but makes it easier for an attacker to retrieve the seed from a targeted individual by compromising their device.

Reidentification

Reidenitifaction is the threat that a postive testing individual can be identified after submitting their interactions. While no previously discussed method can completely eliminate this threat, there are several ways to mitigate it.

In the Bluetooth models, a user may upload their temporary keys to a public health authority. With the Exposure API method, this information is available to all users, who's devices then locally check it against a list of messages that they have received, alerting them if there has been a match. Reidentification can happen if a third party can use this information to identify who that specific person is. If a person only had contact with 1 person on a given day for instance, they could effortlessly extrapolate who that contact is.

The "PACT" whitepaper 4, which is similar to the Exposure API, describes how the app itself, may withhold additional information such as exact time and date of the encounters, admiting that this is only a mild mitigation as sophisticated users would still be able to retrieve the exact messages from their device. In the centralized model, instead of publishing all encounters, the public health authority may choose to ignore certain encounters deemed irrelevant or withhold or fuzz the exact time of the encounter from the end user. A "human in the loop" may reach out to potentially exposed individuals directly, eliminating the need to publish those encounters at all. This is possible because the authority, having access to all users seeds, would be able to identify those encountered individuals from the positive testing user's uploaded messages.

Relying on third parties to properly anonymize PII is not itself without risk. A whitepaper by the ACLU described an incident in which South Korean officials, working on a policy to anonymize and publicize location history published a description of a "43 year old man, resident of Now on district" who was "at his work in Mapo district attending a sexual harassment class."

Insider threats and breaches

Once a health authority receives PII about a user, they assume a (often times legal) responsibility to make a best effort to protect that data. Once that information in "in the system" if may be made available to contact tracers, technical engineers, other affiliated agencies and vendors. Each of which may include bad actors. The Apple and Google App stores already limit who they allow to publish Covid related apps. Use of the Exposure API requires even stricter vetting and is generally limited strictly to public health agencies.

In many aspects, safeguarding this data is no different from the traditional safeguarding of PII and can be mitigated by best practices, vetting staff, enforcing the principal of least privilege and through keeping and reviewing logs. Minimizing the amount of data collected and the length it is stored for can also mitigate what bad actors can do. In the case of Covid tracing apps, information about individuals looses its value very quickly and so there may be less of an incentive to keep it indefinitely in a non-anonymized state.

Given the immediate demand for contact tracers it may be infeasible to properly vet all contact tracing candidates. Researchers at George Washington University have estimated that the United States alone may require 100-300 thousand contact tracers 5. In this case, it may actually be more secure to collect more information. The PACT whitepaper proposes "mobile assisted interviews" in which a person may programmatically submit the majority of their PII, of which only some may be shared with the contact tracers directly. Phone numbers may also be obscured to contact tracers who can instead be asked to contact an individual though a supplied application.

Hardware and software threats

In addition to the human element. Any application or device will have the usual technical threat vectors. In the examples described here, the device is typically a smart-phone which must accept arbitrary inputs. There may be a malicious QR code or a specially crafted Bluetooth message intended to cause a buffer overrun or other undesired behavior. The device itself may be compromised by unrelated malware through other apps or physical means. The Bluetooth card or other device hardware itself may also be vulnerable.

Applications may have intentional or unintentional backdoors, either put in by the developers, snuck in by attackers or introduced through vulnerable dependencies. Software may contain bugs and unknown vulnerabilities. These flaws can be mitigated by proper software development lifecycle best practices such as vulnerability scanning and code reviews and can be further improved by open sourcing the code and promoting bug bounty programs.

Conclusion

We live in novel and challenging times, as people and companies rise to the occasion and attempt to help in whatever way they know how. The struggle between security, convenience and effectiveness is nothing new, and neither are many the threat vectors described in this article. As our global population continues to look toward technology, we must make sure that we are doing our best to keep safe the very people we are trying to help.

Footnotes\
1 Unified research on privacy-preserving contact tracing and exposure notification\
2 Bluetrace whitepaper\
3 Quarantined After Waving at Coronavirus Patient: How Accurate Is Israel's 'Terrorist-tracking' Tech?\
4 PACT Whitepaper\
5 Contact Tracing Brief