DEV Community: Rob Sherling

React Native PSA - select and highlight text with custom context menus

Rob Sherling — Mon, 18 Dec 2023 02:04:53 +0000

Image credit - realtime photo of me trying to update this library, thanks to @cookiethepom on unsplash.

Before I begin, huge shout out to the two repositories that this was forked from: @astrocoders with the original, and @alenToma. Without their work, this never would have happened.

What is this?

In short, you can now select text again in React Native, fixing a six year old issue.

You can select any text on iOS or Android and give custom context menu actions that you define.

In short, you can do this now (clicking play gif will unstretch the image):

Repo

npm

GitHub

Relevant info

Basically, the libraries that this was forked from hadn't been maintained for iOS for years and required hacks and workarounds to even get working. I just removed old dependencies, updated the iOS code to compile correctly, and cleaned up the readme.

I tried to use ChatGPT to help, but honestly it was just analyzing iOS error messages and removing odd JS dependencies one-by-one.

If you see any problems with it, PRs welcome!

iOS Certificates and Profiles in 5 minutes

Rob Sherling — Tue, 14 Dec 2021 00:08:44 +0000

Hi. This is kind of a quick-and-dirty article about how certs and profiles work, because they're boring to read about and hard to understand (for me).

This is my understanding of how they work. This might not be completely accurate, but it's been enough to work with. I'll cover the concept as fast as possile, and then a practical example.

Before we get into it, terms:

cert, certificate = signing certificate
profile = provisioning profile

Let's start.

Certs and Profiles: concept

Certificates

Certificates are basically permission vouchers to build your app for a specific purpose.

We have two major categories of certs: Development and Distribution.

Development is for local debugging (think hooking a device up to your computer via usb).

Distribution is for making non-debug, stand-alone files that are usually used in two ways.
Ad-Hoc: you plan to distribute your app to a few devices outside the app store for testing.
AppStore: Apple will distribute them on the App Store.

For Ad-hoc, you'll almost always use it in the super-early starting phase of your Dev to make sure your app builds correctly and then never again because Testflight is just better.

Profiles

A profile is tied to a specific certificate (again, for development or distribution) and is a permission slip to build an app for a specific set of devices.

Note that a build profile that is tied to an AppStore cert will not ask you to specify devices, because Apple will distribute it to the world at large.

Actual practical example

So, in this example I'll go through development that starts on an iPhone and then goes to iPad.

Step 0) Buy an iPhone.
Step 1) Go into your Apple developer account and click on certificates. Right now we just want to do some dev and testing via USB, so we choose development.
Step 2) Add the iPhone we just bought to our Apple Developer Account list of devices. This is an account-wide list of all the devices you want to use for dev.
Step 3) Go to the profiles section. Click create profile. When it asks you what cert you want this to apply to, click the development cert we just made. It will ask what devices you want the profile to allow installation for - click your iPhone. Done.

Now we can use this cert and profile to do some dev. After a while, we realize that we want to test on iPad as well.

Step 4) Buy an iPad.
Step 5) Go back into your Apple Developer Account and add the iPad to your devices list.
Step 6) Create a NEW profile. We have to make a new one because we cannot edit old profiles to add devices to them. The new profile will ask what cert (our only cert, the dev one) and what devices (check iPhone AND iPad). Done.

Now we can do dev on both.

After a while, we want to upload to the app store. Good.

Go to the Apple Dev Portal, create a cert, but this time choose distribution - App Store.

Then create a new profile and at the cert step, choose the cert we just created.

Done.

I hope this was helpful.

My AWS service was over-provisioned by ~1000 times: A 5-minute guide on how to check app resource use on fargate

Rob Sherling — Sat, 09 Oct 2021 20:30:38 +0000

cover image by @marcusWrinkler

Very quick backstory:

I built and am working on an app called Kichi. It's an app to help intermediate learners continue to learn Japanese and English at a very fast pace.

I've done a lot of deploys in my software career, and I've found a really good recipe for starting any new deploy. In AWS speak, you want to use RDS with a db.t4g.small wired up to two t4g.medium. This makes sure that you can handle unexpected loads if your service suddenly spikes, and that your DB won't go down assuming that your app is more process-intense than it is DB-intense. You then want your DB to be AZ-multi-instanced so that you don't have to worry about an outage taking you offline and so that your maintenance window doesn't take your service offline.

This is an excellent strategy for launching a service for a company. This is a terrible strategy for launching your own service from scratch. For reference - I knew my service would need to start small, so I set my db to micro and my two containers to 1 GiB memory with .5 vCPU.

How to actually profile your production app

So, something that I found profoundly annoying is that everyone seemed to tell me "profile your app to see how many resources you need", but no one actually told me how to do that. So, I found out.

1) Put your app on Fargate. I only say this because this is my current setup.
2) Click "Enable container insights" when you deploy. This will send a ton of metrics to CloudWatch. Note: this is very, very expensive to maintain long term.
3) Use your service for a little bit. Send a few of what you think might be your most demanding requests.
4) Go to the CloudWatch Management Console, click "Logs", then "Log insights".
5) Click 'Select Log Groups' on the top of the screen, and pick your logs that look something like aws/ecs/containerInsights/{clusterName}/performance
6) Prepare to be shocked at how efficient your app probably is.

Please note that for each query, simply change avg to max or vice-versa to see the average or max for that metric.

Memory Usage (in MiB)

stats max(MemoryUtilized) by bin (30m) as period, TaskDefinitionFamily, TaskDefinitionRevision
                | filter Type = "Task" | sort period desc, TaskDefinitionFamily |  limit 10

The number that comes out from this query will be the maximum memory that your containers have actually consumed, in MiB. I had provisioned 1GB per container. My max was ~299, so I dropped it to 512MB per container. Saved 50%.

CPU Usage

stats max(CpuUtilized) by bin (30m) as period, TaskDefinitionFamily, TaskDefinitionRevision
                | filter Type = "Task" | sort period desc, TaskDefinitionFamily |  limit 10

This one blew me away. So, the number that came out of this search was 0.7. I thought "Okay, so that's like 0.7 vCPU, that's not too bad."

Then I ran a different query to see what the container itself had provisioned for maximum CPU usage, just to double-check (that query can be found in the metrics source link at the bottom). The number that came back was 512.

Not .5. 512.

My app, with all of its OCR and DB access and sync and whatnot, only used about 1/100th of a vCPU in production.

I immediately reduced my docker containers from 2 to 1, and reduced the CPU allotment to 256. This saved me 75%. Then I turned off container insights, because CloudWatch was getting expensive and I wouldn't need to profile again for a long time.

So yeah. That's how you actually profile an app and see how much Memory/CPU it takes, and figure out how over-provisioned you are.

Thanks for reading.

Source for all metrics: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html

Working in Japan 2: Freelancing, Job Hopping, More Salary, Visa Types, Learning Japanese

Rob Sherling — Sun, 08 Aug 2021 12:58:18 +0000

Cover photo by https://unsplash.com/@themcny

This is a follow-up to Part one. This is an English article - I will only use Japanese when not using it would make it harder to convey a point.

Disclaimer - In the language section, I'm going to talk about Kichi, an app to help you learn Japanese to an advanced level by helping you read material that you are interested in. I made this app. It's incredible, and I use it every day.

Types of freelance
- Freelance company
- Direct contract
- Companies that aren't freelance but look like it
- No need to network
Benefits over being a regular employee
- Salary differences
- Meritocracy
- Remote work is easier to get
- Fewer meetings
- Easy access to cooler problems
- No code tests
- So much time off
- Flexibility in work hours
- True freedom
- One interview
- Job hopping isn't a big problem
Downsides
- No stability
- No friends
- Language
- Skill
- Benefits
Compensation
- How much to charge?
- Health insurance
- Taxes
Visa - Highly skilled professional visa
Learning Japanese
- Full explanation
- Very Short Version
- An app to help you learn
Conclusion

Types of freelance

Freelance company

Like everything on this list, a freelance company is a mixed bag. They manage invoicing/billing and will arrange interviews on your behalf. In exchange, they take a cut behind the scenes. You typically get paid the next month, e.g. you worked the month of May, and get paid in June. Of course, their cut means less money that can go to you.

The way it works is that they have a bunch of job listings on their site. You create a profile, talk to an agent, and they match you with listings based on your needs.

I've seen some shady freelance companies that were like "we keep your profile on record and forward your information to prospective companies. You get paid after they pay us, so two months from the date of work. Also, we have 'incident insurance' that you can buy. If the company doesn't pay us, you still get paid half. Don't worry, that never happens. But still. We have the insurance".

The company I ended up choosing to help me find freelance jobs wanted to come with me on every interview, which was a bit awkward. Something I didn't realize was that them coming with you on each interview is also a big advantage - they typically give you very honest feedback about what went well or poorly in an interview based on the company's feedback. They get this feedback by asking you to wait outside the interview room right after the interview and get the company's immediate impression of you.

This works because the client company needs to be honest with them about how the interview went so that the freelance company can find better candidates, which will help the freelance company get paid. In turn, the freelance company can pass that feedback onto you so you can be hired more easily which will also help the freelance company get paid. It can help you realize areas you might need to work on or skills you might want to project better.

These types of places always pay you directly month-on-month. You don't need to send an invoice. Just do the work, file a report at the end of each day summarizing what you did (mine were usually about one line of text), and get paid. Simple.

I chose a company called "Crowdtech". I'd recommend them but you'll need some experience and language skill, covered below.

Direct contract

You reach out to a company directly. Use normal job hunting platforms and when you apply just tell them that you want to freelance. It's pretty surprising - a lot of companies are willing to do this because for them it's less risk. A lot of the protections that full-time "regular" employees have don't apply to a freelancer, i.e. they can just not renew your contract if it doesn't work out. Of course, you make more money compared to "salaried" workers because you lose those protections. Caveat - you can't negotiate a starting bonus as a freelancer, so take this into account. A starting bonus is for employees who will work long term, and freelancer implies leaving at any time.

They may pay you directly month-on-month without you doing anything. Alternatively, you may need to send them an invoice on the last of the month and they pay it on salary day the month after. For example, at the end of April, you create an invoice for April's work and give them a copy, and they pay you sometime in late May.

If you're going to send an invoice, use a program called "Misoca." It's a bit tough to register for but super helpful and free. Strongly recommend.

This is nice because there is no middle-man to take a cut of your salary, but it does require marginally more work. Still, this is my recommended method.

Companies that aren't freelance but look like it

Two types of companies: SES and 派遣会社. They hire you full-time and then send you out to different client companies so you can work side-by-side on the client's projects. This is the opposite of freelancing. You don't have a freelance contract, you don't control your freedoms, you stick to the client's schedules and expectations, etc. Would not recommend.

No need to network

A note on finding independent jobs and work on things like Upwork / Crowdworks. It seems like in other countries it is common to build your freelance career by networking and warm introductions and connections. In my experience in Japan, you can just job hunt like everyone else and simply ask to be freelance. You can go the networking route if you'd like, though. I've seen it happen - I've even seen a few freelancing companies that were started that way.

Please note: I don't do bill-by-the-hour freelancing or Upwork or any of that stuff. Bill-by-the-hour work is not as common in Japan as it is in America. If this is the type of freelance that you are interested in, I would recommend googling it.

Benefits over being a regular employee

Salary differences

You make more. Like, much more. It's further down the article, but you're taking on more risk so it makes sense that you make more. The exception here being starting bonuses, as mentioned above.

Meritocracy

When you have a normal "desk job", you have to be in that building for at least 8 hours. With freelancing, If you get the work done, people rarely care what hours you work. I've had freelancing gigs where I worked remote 4 days/week and on my office day I was there from about 9:15 am to lunch. Your time is your own, and it's great.

Remote work is easier to get

Remote has become far more common during the pandemic, but it's easy-peasy to work at least a few days a week from home and just do the office 1/week thing. It's a really big perk of being expected to crank out quality work constantly - no one cares where you are when you do it, as long as it gets done.

Fewer meetings

Double-edged sword. This contributes to a feeling of not being a part of the team, but I will take that trade 10 out of 10 times if it means that I only get roped into essential meetings. Counter-intuitively, I strongly recommend doing a 15-minute stand-up meeting each day if you're mostly remote. It helps people keep track of what you're doing and keeps you in the mind of your project manager, which is important for a lot of political reasons. Plus, it helps you do your job by keeping you abreast of sudden organizational changes and developments.

Easy access to cooler problems

This certainly isn't exclusive to freelancing, but you can get some crazy problems that no one else on a team can deal with, like build tools, profiling, or migrating systems. If the idea of minimal onboarding followed by "We need you to build a prototype in 2 weeks, can you do it?" is exciting, you shouldn't have any issues finding that kind of work. Also a huge upshot - as a freelancer I never get assigned to routine bug fixes or maintenance code, because we're expensive and prone to change jobs often. Interesting, hard problems are in everyone's best interest.

No code tests

I like code tests (think whiteboard interviews, not hacker rank) but it turns out that according to Japanese law you cannot be asked to take a code test if you are a freelancer. Polish those side projects if you don't have a large work history, because they can't directly test your aptitude so they'll need to see something.

So much time off

If you finish the work, it's stupid easy to take days off. These are often not paid days off, but most freelance contracts are written like "service provider will work between 140 and 180 hours per month." If you get all of your work done, 140 hours is a little more than 2 days off / month. Also, if you work well with the company, most companies will still pay you a full salary even if you take a week off once in a while. Seriously. It's the wildest thing, and I can't explain it, but I've had it happen at more than one company where they told me "If you make the hours back here and there, we aren't worried about a week."

Another big advantage is that taking unpaid time off as a regular employee can damage relations with your company. Not so for a freelancer - all time off is unpaid, so no worries.

Flexibility in work hours

This goes hand-in-hand with everything else, but screw a 9-6 workday. If you work best in the morning, put in a few hours first thing, enjoy your day, and then finish up in the evening or at night. It's up to you to make sure you get the work done and fulfill your contract hours, of course, but I've worked 5-hour days for a couple of days a week and finished out the time on a Saturday where I had no plans. You have infinity-flextime - use it.

True freedom

All of the above means that you get to experience the absolute best life that engineering can offer (in my incredibly humble opinion). If you want to work from another country (a little overrated, actually, but still nice!) or travel from Tuesday to Thursday because tickets to Wakayama were dirt cheap and that Airbnb with the rooftop pool lounge was only like 7,000 yen a night, but you don't want to suffer from bad work karma for taking time off? This is it. I despise being chained to a desk, and freelance is giving me everything I could ever want.

One interview

Most freelancing gigs are one or two interviews, but it isn't uncommon to just have one 45-minute interview and then be hired. Lower risks for the company means faster turn-around. Regular employee interviews can also be short, but I find that freelancing interviews on average tend to be shorter.

Job hopping isn't a big problem

Changing jobs often as a freelancer, especially if you have fulfilled your current project, is usually fine. The conventional wisdom in Japan is 3 years of employment. I usually change jobs 1/year, so freelancing is perfect for me.

Downsides

If it isn't clear that I'm very biased in favor of freelancing, I am. But there are some real downsides.

No stability

In short - Your contract can just be gone if they opt not to renew.

Most contracts are one-to-three month automatic renewing, but the details depend on the contract. That being said, I think because of the way these things are handled in Japan you are more likely to have a warning before your contract suddenly ends - my last freelance company had no more work for me but extended the contract by a month to let me start job hunting because they only realized about a week before my contract ran out. They kept me on longer so I could job hunt from a place of security. I'm still good friends with the people there by the way - amazing place.

No friends

This has more to do with the remote nature of freelance work compared to freelancing per-se, but you get to skip out on a lot of the meetings because you aren't part of the regular culture there. You also don't get access to certain slack channels because of company security reasons. That means missing out on social interactions with your coworkers. You can certainly make friends, but you don't interact socially with people as much if you're aiming for a full-remote type of thing. If you tend to change jobs more often you also won't form a lot of deeper bonds.

Language

I am going to talk about this extensively below in language learning, but in short - if you don't speak Japanese to a comfortable conversational level, you're going to have a really hard time getting hired as a freelancer. There are opportunities out there, of course, but you need to be able to demonstrate that you can take tasks and work them out without miscommunication. A company isn't going to work with you and wait for you to grow your Japanese ability as you work together because you have no long-term relationship. They hire you for your skills now, not what they can grow you into over the next 3 years. This brings me to my next point:

Skill

You need to have previous job experience and skills. In my experience, most people won't consider your first job for freelance even with an insane portfolio. Job experience tends to gets you the interview while skills will keep the contract renewing. Because companies aren't interested in "growing you", if you don't have skills you won't be hired. A big part of that salary bump is that they aren't investing other resources in you.

Benefits

If you are going to have a child, strongly consider becoming a regular salaried employee (the word in Japanese is 正社員). After one year of salaried employment, you are eligible for 1 year of paid maternity or paternity leave. You also become eligible for paid leave if you have a disability that stops you from working. The amount of time for disability leave can be based on your circumstances, so I'd encourage looking it up if you think it might apply to you.

You get paid leave when someone close to you dies, you usually get more money based on family conditions (for example, if you have kids, your salary goes up by 10,000 yen/month per kid), and it's very, very hard to fire you.

Another big benefit is for apartment hunting and taking out loans / getting credit. Big, stable companies especially look great on any kind of credit-based application.

Compensation

How much to charge?

TLDR: Your normal salary + 30%.

Whatever you want to make normally, add about 30%. For example, I usually charge about 750,000 yen/month for my work, so for freelance I tend to charge about 1,000,000 yen/month depending on the project and the time I need to put in. Keep in mind that if you are going through a contracting company they take a cut, so you might not get the exact salary bump you're hoping for. Also, if your output doesn't justify your salary, you are far more likely to be fired instead of the company working to improve your performance, so make sure you deliver that value consistently.

Health insurance

Go to city hall whenever you want, apply for health insurance, pay pretty much the same amount as a regular employee, get the same affordable coverage. It's awesome. America should try this sometime ;P (cue angry comments)

Taxes

This could be an entire article, but they're surprisingly straightforward.
How much money did you get paid all year? Revenue. What did you spend on equipment and train commutes? Expenses.

e.g. You work from home? Part of your internet bill, the electric bill, water, gas, and the portion of your apartment that is a dedicated office are all expenses. Expense everything that's reasonable. Eat lunch with your coworkers? Business lunch. Expensed. Go out drinking as part of a work event? Expensed. Keep receipts.

From year two onward, they'll pre-collect 2/3 of your estimated taxes as two separate payments made throughout the year. At tax time, you take the calculated expenses, any pre-paid taxes, and your revenue. Go down to the tax office, punch some buttons on a computer, and it gives you a number that you either pay (most likely) or get deposited into your bank account (less likely). You can even do the whole thing from home from your second year onward and just bring in the finished forms.

When in doubt, grab all your paperwork and go to your local tax office. They are shockingly helpful.

Bonus #1 - ふるさと納税

The scope is too big for this article, but google ふるさと納税。 Prepay your taxes (to a limit determined by last year's income), get crab/cake/strawberries/wagyu for free. For example, last year I got like 4 kilos of crab and I only had to pay the taxes I would normally pay + a 2000 yen flat fee for using the ふるさと納税 system. This isn't even a freelancer exclusive thing, but you should definitely do it if you are a freelancer. So much free stuff for the taxes you were going to pay anyway.

Bonus #2 - American? Read this

Global income and taxes as an American, the ultra-abridged version. Google what you don't understand.

This is the one place that it sucks to be a freelancer in a weird way - you get taxed on your revenue in Japan, and as a freelancer, your revenue is higher. Hence, you pay more - whereas in a company many benefits are taken into consideration before you are paid, so you make less but enjoy a lot of perks.

Federal tax: You MUST file every year, full stop, freelancer or not. File form 2555. If you make less than ~$100k in Japan, take the FEIE so you owe nothing in tax. Follow the steps, remain calm, and you'll be fine. It's not too complicated in most places. In certain situations (you make more than 100k in Japan), you might want to instead take a tax credit for the tax you already paid in Japan. That's too hard to put into the abridged version, so get ready to read a lot of paperwork.

If your rent is more than (I think) $1400/mo, you can deduct some of it from your American taxes depending on where you live in Japan. Check out the foreign housing deduction/exclusion.

If you have more than $10k (combined) at any point in the year in all of your Japanese accounts, file an FBAR. If you don't, the penalties will rip you in half. This also triggers if you have $5k in one account and then move it to another account.

Lastly, some states don't charge you state taxes if you live overseas. I am fortunate in that New York does not. Note to residents of California - they seem to charge you state tax on your overseas income the vast majority of the time. You should look into that.

Visa - Highly skilled professional visa

Explanation

You get 5-year visa renewals, fast tracked permanent residence, and your spouse can work full time instead of just part-time. You can bring a domestic helper from your country if your spouse works, is ill and "can't do housework", or if you have kids. It's on a point-based system, but in short: Get the JLPT N1, have a bachelor's degree or higher, make a bunch of money, and you'll probably qualify.

The huge catch: In this specific case, the company you work for owns the visa, not you. If you change companies, you lose the visa. This is completely different from a normal visa. I would not recommend it for freelance workers.

Normal work Visa

Many companies have no issue sponsoring you as a freelancer to renew your visa. They might not want to sponsor you to come to Japan to work (few companies do), but renewing tends to be very easy.

Learning Japanese

Full explanation

If you are interested in my take on how to learn Japanese and how I did it, please read this:

How I learned Japanese to N1 as a translator and software engineer. If you like that article please clap a bunch for it because I have no idea how Medium works.

Very Short Version

This is the omega TLDR of the above article:

What level of Japanese should I have?

I recommend ~JLPT N2.

Why?

That's the level that your conversation really starts to flow. Read the above to learn more.

Do I need to know kanji / hiragana / katakana?

Yes. You should be able to read and type all of the standard kanji, more or less. Note: not write, no one cares if you can't remember how to write the kanji for 納税 by hand.

How should I study?

In short - 1) have fun, 2) immerse when you can, 3) review religiously but delete flashcards you don't need anymore as if they are costing you money to keep around.

I still make a lot of mistakes when speaking - is that going to be an issue?

Usually, no. If people can understand what you want to say, you're probably fine. Flip the script here - if someone was talking to you in English and said "I am wanting to go to the lunch together, let's eat some roast foods!", you might find that grammar endearing, and it certainly wouldn't stop you from going to lunch. The inverse, however, "The server is down because static page on CDN was eaten by the node." -> Not being able to understand what this person is saying would be intensely frustrating. In short: if you know how to make yourself understood and are comfortable learning specialty words for your job as you go, no problem.

An app to help you learn

I covered this in my language article above, so I won't plug it too much here - I built a language learning app that helps you learn Japanese from things you're interested in. I used Anki to do this, but I got frustrated with some parts of Anki and wanted to do some things differently. You take images of things you like (menus, games, subtitles, physical books, screenshots of the kindle app, etc.) and it processes them. Tap a word in the image and it will give you the pronunciation, translation, more lookup options, and use the sentence you learned it from as an example sentence in a flashcard. It is specifically designed to help you customize your study to read the things you find fun instead of pre-made decks. I use it to read boatloads of Manga and kindle books, and sometimes play Japanese games.

If you're interested, check out https://learnkichi.com.

Conclusion

Everything here is just based on my own experiences. Every situation is different - some people really, really prefer the structure and security of salaried work, and there is nothing wrong with that.

Did I miss anything? I feel like there isn't a lot of information like this online, and I'm trying my best to post relevant updates. If you want to see an article on something, please let me know. You can leave a comment, track down my email, or just DM / follow me on twitter (link in my profile).

Again, if you want to know more, Part one is here.

I'm a programmer changing jobs in Japan. This is a list of all the stuff I've learned on my job hunts.

Rob Sherling — Wed, 12 Feb 2020 11:52:55 +0000

Cover photo by https://unsplash.com/@gk3

Before we begin - never call yourself a programmer. Software engineer would have made my title too long for Dev.to to allow, so I used shorthand.

Okay. Let's talk about interviewing. This is a collection of advice and useful questions that I've personally collected. If you're in a rush and only want the TL;DR, read the bolded headlines.

I've already written up a really extensive history of my career, complete with past salaries, some negotiation tactics, and a discussion of Japanese work culture here. I'll be skipping all that to save time along with all the obvious interview questions that I see like "what does a typical day here look like?" - that information is everywhere.

These views will be based on me and what I value - every situation is different, but everyone should get good mileage out of the interview questions at the very least because they grant insight into the place you might be working.

Questions

Question set #1: What do you, personally, like the most about the company? What do you want to improve the most? (do not use the word dislike)

Most people give very safe answers to this question. This question serves two really good purposes, though.

If they give a genuine answer (huge bonus points from me, by the way) that isn't immediately obvious, I ask them to tell me why they feel that way so I can learn more about the company as well as the interviewer who I will probably see again at some point in my life should I chose to work there.
I forgot literally every single question I had prepared / they were all answered and I want to seem unorthodox and insightful.

If you ask what they like the most, they should give a genuine answer. "I like that we have good communication and work as a team / I like that we have a new coffee machine that does cappuccino and it's become the morning hangout spot / I like that we've got a very direct work culture." This is a fairly easy thing to answer, and I mostly gloss over the reply unless the answer is actually interesting.

When you ask what they like the least, this is where you learn about them. Do they give safe answers like "I wish we had more people"? That doesn't necessarily count against them, but it says more about the company culture if they can say (especially in front of their teammates) "We're working on being more honest and more forthcoming with our opinions so we can work more efficiently / We desperately need more people in leadership positions because we need someone to guide the less experienced devs / We got our overtime work down to an hour a day, but we want that to be even less". If you hear this, that is an excellent sign of honesty and wanting to genuinely improve the company. Huge plus.

Write down the positive point if you find it even remotely interesting. Before the next interview, take 10 seconds to look at your notes. If you're going to see that person again, it's impressive to say "Oh, John, I remember you telling me about the coffee machine! You do a cappuccino talk session today?". They will love you if you do this in a natural way.

Write down the negative points in your notes so you can come back next interview and talk or demonstrate in a round-about way about how great you would be at solving those things and how honest/communicative/not-overtimey you are.

Question #2: How many people have quit in the last year / what percentage of people quit this year?

Super small Japanese note because I found this super useful, the word you want here in Japanese is 離職率(rishokuritsu).

This number is incredibly important. Whether you should ask for a percentage or just a total number depends on your culture, but if they tell you a number that seems high, you should be concerned. I don't know what it's like in other parts of the world, but in Japan, for engineering anything higher than 5% is a problem.

I found out once by asking this question that the entire engineering team for a startup had at one point all quit. Literally every single person left, in the span of like two weeks. Ask the question. If you're afraid to ask, that says volumes about the work culture and you don't want to be there.

Question #3: What are your hobbies?

This is nice because it is polite conversation that isn't work focused. You get to see their passion for something outside of work, and you might get to see what their work-life balance is like. If they don't come up with anything (some people don't), no pressure. Just move on smoothly, or wait for the rebound and talk about something you like to do outside of work.

Question #4: Do you have a process for meetings?

I ask this for bigger places. What are your meetings like? Are there a lot of them? Do you like them? Is there a rule on how long one can go on? Do the engineers go to them often? Etc.

Because meetings usually suck. They don't have to - if everyone in a meeting room is arguing / laughing / getting angry / brainstorming, that meeting is worth having. If no one has said anything of substance in two minutes, or if the meeting could have been an email, you're wasting your time.

This also, if you're lucky, can lend some insight into how fast the company moves and how much bureaucracy they have. A company with a lot of meetings rarely is process-free in other areas.

Question #5: When is the code interview?

Ask for a code interview. It does two things:

1) You get to show off your skills if you know something, and learn a new type of problem if you don't.
2) It shows you what they value in their engineers.

If they ask you to architect a system or design a database for a realistic use-case, you are in good shape. If they ask you to do a quick sanity-test for coding like Fibonacci numbers, that's fine too if you're more junior / they scanned your Github and just need to check that you actually wrote that stuff.

You are not in good shape if they ask you:

A math question disguised as a programming question.
A pointless, easily googleable question about syntax like What’s the difference between varchar and varchar2 in Oracle 8i
A general algorithms question. Hear me out on this one: If you are working for a place that requires novel solutions and original code, good. This is a valid question. Otherwise - if you won't use this in your day-to-day, why are you being asked it?
An abstract question like "How many stop lights are there in Seattle?" to see how you think.

I am sure that you will find avid defenders of the above four questions in the comments, especially of that algorithms point. I stand by my views.

Question #6: What did you do before this, and why did you change?

Useful for building rapport and general conversation with the interviewer. Also, if you care about career advancement, there's a bonus to asking this: The answer often tells you if they got promoted to that position or hired in as a manager already.

Advice

Advice #1: Do not write most things down.

I'm serious. Spend most of your time just listening because eye contact, genuine expression, and communication are infintely better than extensive notes. Only write down answers to questions about things that you care about, things you forget easily that you'd wish you hadn't (names, for me), and impressions of the company and people.

I've found that writing down "seems overly serious at first, but actually really funny" helps me remember that person's personality a little more accurately later on. I've also found that it helps confirm any good or bad vibes that I had when I go in for another interview later. When I look at my notes and see something like "This engineering manager seems like they are painfully strict and don't respect their employee's free time" from last week, followed by "this HR manager seems like they don't care about their employee's time", that's some pretty hard evidence that I should stop there.

You can always send a followup email asking about a particular process or system or something that you forgot about or didn't write down. It's much harder to recall a hiring manager's attitude on work-life balance a few days after the interview, much less the week after when you're trying to decide what upcoming thing to cut from your schedule because you realized that smashing yes on each casual interview offer was a terrible idea.

Advice #2: Don't smash yes on each casual interview offer.

Note: This is not saying that you shouldn't look into each interview offer, just don't smash yes.

I was super, super guilty of this. I would get a bunch of offers to "come to the office to chat about the position", took a bunch of them, got there, and this would happen like clockwork:

15 minutes of conversation
I ask what the salary range is / overtime is like
They answer with 1/2 my current salary / 20 hours a week
I now have an hour and 45 minutes until my next interview, so I get to go pay Starbucks money again so I can sit there and wait until I can repeat this process.

It costs a company nothing to have you come to their office for one of these. If the casual interview goes badly, they just go back to work. You lost all that time to get there and do all the research to make it seem like you knew a bunch about their company.

I actually went to one casual interview to have the interviewer ask me if I had big data experience at a Facebook or Google scale. When I told him that I had not, in fact, worked at a company even close to that size, he was surprised. I think that if I had actually read the job description (and if he had read my resume), we would both have saved a lot of time. We talked about Russian face-recognition technology, which was kind of cool, but we could have done that over the phone.

If you see a place reach out to you that wants 3 years of front end experience but, like me, you would rather drink a mild poison than work with CSS, think about it. I'm not saying to skip the opportunity, but absolutely send them a message first. Mine is almost always:

"Hi! Thanks for sending me an invitation to come talk. I have a concern that I'd like to address first, if possible. You said that you specialize in X technology, but I've literally never heard of that before and don't even know how it's pronounced. I'm more of a Y technology type of person - is that still a good fit?"

Half the time, no response. Half the time, "Actually, that's an old job ad / a job for a different position, but we saw your skill set it matches something we have opening up."

Rarely, "You look like you have a lot of experience - I'm sure you can get good with X technology quickly, so I'm not worried."

Go to those.

Advice #3: Never talk salary with anyone who isn't HR/manager at final stage

I realize this might be obvious, but trying to talk salary with another engineer or someone who can't decide in an interview is usually not a good idea (and I have done this). Focus on the fit, then the money. If a manager without salary-deciding power comes out of an interview and says to HR "they wanted to know how much we can pay", it does not come off as "This person wants to work with us."

Advice #4: Salary discussion in the first interview should raise suspicion.

You asking for numbers in the first interview can be totally normal if that's your negotiation strategy. I do it occasionally to get a range of acceptable salary. If they ask you for your expected salary, they are not doing you a favor, they are trying to determine how much you'll cost before even seeing if you have what they want.

Let me put it this way - if I told you I was going to buy you a mysterious fruit that I had never seen before from an old man in a wizard robe on the street, your first question would not be "How much does it cost?" because there is so much more that needs to go into that decision before price even becomes a factor. People who decide on price at that stage are not people you want to be with.

That being said, if they ask and you respond with "what's the range of salary?" and they tell you, no problem. If they insist/are pushy, walk.

Advice #5: The questions they don't answer are the most important.

If you get to a company that answers every question you ask, or follows up if they don't have that info on hand, it is probably a great place to work if you value honesty and communication. If they don't answer the question, assume the worst. Sometimes this doesn't mean anything, sometimes it's incredibly important.

Examples:

"How many engineers are working on your level 3 customer support team right now?"

"Can't say, sorry."

Completely fine, probably a business secret or something.

"You said that you're looking to grow a lot this year. How many engineers are you planning on hiring?"

"Can't say, sorry."

That shouldn't really be guarded information. Not a red flag by itself, but pay close attention and double check to make sure the company actually involves engineers in the business decisions.

"How many people have quit in the past year?" "How much runway do you have left?"

"Can't say, sorry."

Walk.

Advice #6: Imagine being a Toxic Person

Okay, weird title, but I'm serious.

If I'm going to work with a company, I am painfully honest and try to communicate as much as possible before the hiring stage. But I also try to think of the hiring process from the mind of someone who would have a toxic affect on the work environment and see if they've done anything that would have weeded me out.

If they talk a lot and don't let me talk much or ask many questions - I could be skilled but not a good person to be around. If I'm not introduced to anyone on the engineering team, they won't know if I have a clashing personality with our front-end engineer.

The reason this is important, of course, is because you want to know that the process is good so that when you join it is less likely that you will have to deal with a toxic person joining after you.

Advice #7: Pay attention to your gut.

If you feel off about a single person that you interview with, strongly consider walking. Like a relationship - you have a few hours to decide maybe years of your life. Be smart about it.

If you feel like a company's atmosphere is too formal or casual, or the engineering area looks like a sterile hamster cage, or the sales people look like they'd rather be dead than at work, consider passing on it.

You get a bad gut feeling, walk.

Misc:
When you see a job advertisement, that's actually the first chance you get to see what that company chooses to present to potential hires. If you see some copy-paste HR spiel (The successful candidate will have...), you're going to be working at a place with a very different vibe than a place that writes job ads like Google does. Prioritize applications accordingly.

Once, in an interview, I went in to find the CTO who was supposed to be interviewing me was going to be late. Not a problem, it happens. He came in later, picked up my resume that the HR person had left on the table in front of him, casually flipped through it like I would flip through a pamphlet on the benefits of pistachio water, and then asked me to introduce myself and all my skills.

Then he asked me about part of the Unity game I had been working on - specifically, sending data over a network. I told him, then he told me that couldn't be accurate. I told him it was. He then told me that TCP/UDP cannot be used over phones and asked me how the Unity game engine implemented networking on a fundamental level. I told him truthfully that I had no idea, but I was pretty sure that the online multiplayer realtime rpg I was working on had UDP. He did not believe me. He later went on to say that everyone in the company, regardless of level or performance, received the same bonus amount and suggested that if my performance did not meet expectations they would decrease my salary until it did.

They actually gave me an offer. I did not take it.

Really, if I had to sum this article up:

Company culture is everything. Ask the questions that give you the information you need to decide in two hours if you want to spend 8 hours a day working with these people.

I think I'm going to write an article about freelance/consulting vs. full time work (at least here, in Japan) and how my current job hunt is going. It might be after I've chosen where I'm going to work, so a few weeks at shortest. It will hopefully be shorter than this article. If there's anything you'd like me to talk about, mention it below. I have no idea what people want to hear about and I'd like to help.

Thank you for reading.

Working in Japan: Myths, Realities, Salary, Culture (By A Software Engineer)

Rob Sherling — Wed, 28 Aug 2019 05:56:24 +0000

Cover photo by https://unsplash.com/@manucosen

Edit (Aug. 5th, 2021): I wrote some new articles about how I learned Japanese and my language learning app Kichi. You can read about how I learned Japanese here.

Programming in Japan

Intro
Qualifications
Starting My Career Here
Working Conditions
- Hours
- Paid Leave
- Contracts and Rules
- Working in Games
- Company Rigidity and Change
Employment Overall
- I Was "Let Go" Once
- Excellent Job Security, But Not Guaranteed
- Bad Bosses, Harassment, Options
The Really Good Parts!
Japanese Attitudes Towards People Who Are Not Japanese
Overtime
Interviews and Perception of Programmers
- Attitudes Towards Programmers
- Interviews, Competition, Company Size
- Working With Recruiters
- Freelancing
- Jobs That You Can Do Without Japanese
- Money, Visa, Location
- Salary and Cost of Living
- Apartments
Conclusion

Intro

This is a massive, nearly 30-minute read. If you're just interested in salary, click the indexed link above. I promise the whole thing is a great read, though.

This last year working in Japan has been a wild one. I wanted to take the time to offer the perspective of someone who's working successfully in Japan as a software engineer in mostly-or-entirely Japanese companies. Also, I want to give a perspective that hopefully offers some contrast against the mountain of horribly misinformed comments I see regurgitated on social media (looking directly at you, Reddit).

I'm writing the article that I wish I had going into all this - how hard is it to get that first programming job? What do I need to do to make it happen? How do I become a freelancer, and why would I do that? How much - exactly - does one make?

To that end, I have year-by-year breakdowns of my salary, a brief background of my pre-engineering start here, and much more.

A few things before we get started: I have mostly self-taught Japanese. I am fluent in Japanese and have native English. I am a very confident person. I am comfortable negotiating and saying no. I have never worked for a non-Japanese company in a career capacity. All of the companies that I've worked for were overwhelmingly Japanese in both nationality and language.

I'll do my best to list the upsides and downsides of my own experiences, but as always every situation is different. Now that we're prepared for the wave of "but your experience is not normal because X" comments, let's do this!

Also, this post will contain no Japanese. While I can understand the choice to sprinkle Japanese into a post about Japan, I think it would make this post harder to understand without adding anything of value.

Qualifications

As of this article, I have about 4 years of experience as a programmer - mainly back end. Experience matters, of course, but what you do when you're gaining that experience matters more.

I did not learn to program in school or boot camp. I learned to program by reading books on the fundamentals (what is an int, stack vs. heap, etc.) and then just doing a lot of things.
As I mentioned above, I'm very comfortable with Japanese.

Starting My Career Here

I came over as a Human Biology grad with no programming experience and pretty good Japanese (JLPT ~N3 for those who know the test). I could read some utility bills, some letters, and documents, and get by just fine with no English.

I came over as an English teacher. I decided I wanted to be a games programmer, bought a physical copy of programming for dummies, and read on the 40-minute trains between my house and whatever school I was teaching at while I was situated up in Sendai (north-east Japan).

I was worried about telling people that I was an English teacher when I changed into programming. I was concerned that they wouldn't take me seriously, or that it would somehow diminish my credibility. There is a perception with some truth to it that many people come to teach English here because it's an easy, paid way to experience Japan.

I was wrong. If anything, it deepened my peer's respect for me because I was learning something outside of my field, and there's a pretty healthy respect for teachers here anyway.

I studied Japanese and programming intensely for about two years. I got my N1, released a game, and then applied for a programming job. In retrospect, I should have done things pretty differently.

I put way, way too much value on the N1. For those who don't know, JLPT is a test of your Japanese ability. Like all tests, it can be gamed, and it can be very flawed. I thought that I would need the N1 (highest level) to be viable in the Japanese workforce. The level before that, the N2, was straightforward and didn't require much study apart from grammar. The N1 was just all study. So much boring, grindy study.

Day of the test, I fell asleep in the listening and woke up 10 questions later with someone talking about selling lemon-flavored baked goods. I went with the tried-and-true method of choosing all "C" for the questions I missed.

I passed by four points.

Then I got into the workforce and used almost none of what I learned after the N2. I should have just focused more on programming from there.

Also, entering the workforce months sooner would have been wise. I released a small, fairly well-polished game and then started studying Ruby for a few months. My Ruby study had very little to do with me getting hired. I could have gotten hired right after my release.

If I can do it, you can too. Just put the work in.

Edit (Aug. 5th, 20201): Check here for more about language learning.

Working Conditions

Hours

Just like in America, the conditions are as varied as the jobs. Japan can offer jobs that follow 10-7 work hours with an hour lunch - I would consider this the normal schedule in the tech scene. You can certainly find 9-6 jobs, jobs with flex time and core working hours, etc. Meritocracies are rare but exist - often (but not always!) when a contract stipulates that you can work whatever hours you want, it's used to have you work horrific amounts of overtime. I have worked in places where they were willing to change my contract to a meritocratic one. They exist, and the system can be excellent - work hard, go home early.

Paid Leave

Like pretty much every first-world country except the states, Japan has mandatory holidays. 10 days a year, minimum, with a bunch of public holidays. It is true that many companies in Japan make you feel bad for taking paid time off, but from what I've directly experienced this is often less about a boss giving you flak and more about not wanting to abandon your team - a sentiment I've seen echoed in other countries as well. This seems to be less common in the tech scene - competition here is so fierce that you really can't risk losing programmers to that kind of thing, and the government cracks down very hard on stopping people from taking days off. On a personal note, I have never had an issue with it.

Contracts And Rules

Contracts are really, really short. Usually a single page, and very loose.

You have a few kinds of contracts. I'm not going to get into the details here, but basically, you either have contracts that renew and ones that are just permanent. Make sure you understand exactly what is in your contract - it's usually just boilerplate. Side note - check if they pay your taxes from your check or if you're expected to set aside money to pay them yourself. The tax issue is a bit complicated, so just ask.

Similarly, all companies are required to have rules of operation for their employees. If you are unclear on anything, ask to see the rule of operation governing it.

Working in Games

I do not recommend working in games. The entire state of working in games is a sham globally - low pay, bad conditions, churn - but I can only comment on my experience in Japan.

Working in games in Japan is probably pretty similar to working in games in America. The pay is usually lower than in other fields. You can easily expect something south of 300k JPY / 3k USD a month. Not bad if you're totally inexperienced, but the raises and the salary cap are terrible. Ironically, if you want to draw a really good salary in this field, become specialized in either infrastructure or backend systems, then enter the field - you'll easily make double. Again, just my observations.

I worked for a games company in Osaka called Aiming. I wasn't a fan of most of the management, but the people were gold. We had a game area in the company where people gamed during lunch. Fridays, we'd bring alcohol and snacks, drink, go drink somewhere else, come back and drink, etc. - there were, of course, plenty that didn't drink and it was always a great time.

I also interviewed at a few different game companies. One, let's call them Schmooby-Loft, had a particularly interesting interview process. At one point a gentleman from HR made it clear to me that I was technically entitled to take my time off when I pleased under Japanese law. He also made it very, explicitly, painfully clear that it would also almost certainly "be reflected" on my performance review and salary adjustments if I chose to take time off during busy periods.

Not that it's taking time off during busy periods should be encouraged. But they made a show of conveying that in the most intimidating way they could.

I turned down their flattering offer of "my current salary at the time but with more overtime."

Company Rigidity and Change

Japanese companies are famously bureaucratic. However, you can make a massive difference at your company if you chose the right one.

I've changed my pay, my working hours, my team, and my job tasks just by having conversations. Much like anywhere else, it depends on the people you work with and your ability to be persuasive - in this case, in a language that you may not have grown up with.

My approach is always the same - find out who to talk to, and tell them what I need. Managers are usually more than willing to accommodate you if you ask - often the problem is them not knowing what you need.

Employment Overall

I Was "Let Go" Once

I've been let go from one job. I'm gonna be honest - I technically chose to leave, but it was a dark situation to be in.

When I was working at a pretty famous (in Japan) programming company, I had a medical issue. I won't get into the specifics, but the company that I was working for initially seemed supportive. I got medical leave, was put on government-paid benefits while we worked things out, etc.

Unfortunately, after it became clear that I wouldn't be able to return to work for a while, my company decided to let me go. That's not the part that upsets me - I understand their position, I was still eligible for medical assistance, etc.

The part that upsets me is that they asked to talk directly to my doctor to confirm my situation despite my showing them appropriate medical documentation including a certified letter from my doctor. After my doctor said (rightfully so) that that was insane and illegal, they reduced my pay citing work performance - while I was on medical leave. If I returned to work before my medical situation was resolved, and had to take leave again, my new medical leave benefits would have been at the reduced rate.

They then asked me to leave. I did.

I want to make it clear - there's a lot of nuance to this situation and this short description doesn't do it justice, but it does segue nicely into:

Excellent Job Security, But Not Guaranteed

In Japan, it's almost impossible to fire someone. Company restructuring, a flagrant violation of valid company policy, or gross incompetence. That's about it.

In reality, one of the following will happen - and I have seen some happen to people I know- in lieu of being fired.

1) You will be pressured into leaving. This takes the form of management putting stress on you in any way they can until you quit. If this happens, leave. Not because you're wrong - because you're in demand and there are dozens of companies lined up to hire you, so why work for shitty people that do this?

For clarity - I have had an acquaintance of mine in HR talk casually to me about how they were actively doing this to someone. It is very, very real.

2) Restructured. Not a lot to be said about this - if they downsize, which is pretty hard to do here, you can get downsized too. Not very common in tech, and you get paid unemployment anyway.

3) Assigned meaningless jobs. I've never seen this happen in tech, but I'm aware of it in other fields. Boredom until you leave.

4) Contract not renewed - this one's kind of interesting. Your contract isn't renewed citing some reason. If you think it's bogus, you can fight for damages. That's about it, the rest depends on your circumstances.

Bad Bosses, Harassment, Options

In America, I think it would be safe to say that a lot - the majority, even - of the workforce has worked for a boss that they did not think was a good boss. Some have even unfortunately - in one form or another - experienced harassment. In this regard, Japan is no different.

I have worked under several kinds of bosses, in several kinds of companies. I have worked for people who are hands-off, micromanagers, powerful leaders, and people that I believe have no business being in an office, much less leading. I have worked for bosses that inspire me to be a better person, and bosses that inspire me to stay home.

I want to talk about something more specific here, something that I've experienced once to a pretty serious degree - harassment.

Japanese law defines a lot of different kinds of harassment. Power harassment, mental harassment, sexual harassment, etc., but the takeaway is this - there are a lot of ways to harass someone, and the government comes down on them all very seriously. A company can get into serious trouble for inadequately investigating a single claim of harassment.

I don't know how my experience would be different if I were a native Japanese person or not a man. All I can offer is what I've learned from my own experiences.

If your boss does something you think is harassing, for example:

talks to you differently than they do to your peers, but in a negative way
assigns you impossible or disproportionate workloads
sets specific rules that only apply to you
ever is physical with you in any way that you find uncomfortable
undermines you in front of your peers
excludes you from social events

Bring it up to your CEO or HR, whichever you feel more of a connection to. You have a ton of power and resources here, and the government takes these claims extremely seriously.

In my own experiences, I have had two things that I can describe as harassment. One was a drunk boss - who I was very casual and good friends with - smack the back of my head in a disciplinary fashion in a bar. He came to his senses in a few seconds and apologized profusely. I don't think it's an accurate reflection of his character and chose to let it go.

The second one was much longer, and concerned a manager who went from not liking me to outright hostility. I did what anyone in that situation should do - I documented it extensively and talked to the CEO one-on-one. The CEO was an outstanding person, got directly involved, and did his best to reel the manager in. Unfortunately, it didn't work well, and I chose to leave the company. The CEO said that the situation was unacceptable and apologized to me personally for the situation. Again, the circumstances are way more complicated than a short paragraph can do justice.

I consider my experience with this type of harassment to be both unusual and well handled. The CEO was willing to pursue the matter further but I chose to leave because it wasn't worth it to me. There are way too many job opportunities and great companies to have to work with terrible people like that manager. Life's too short.

The Really Good Parts!

We just covered a lot of negative things - and we're going to cover more - but I want to switch it up.

I think exceptional cases make for better reading. I think that negative press gets more attention. But overall my time in the tech sector in Japan (and Japan overall) has been good. Aggressive pay raises on job change, a ton of job security, energized teams, and fun people. Like anywhere, the company culture defines how much you're going to like your job, but I've worked with people and at places that I had a blast almost every day at work.

Working in tech was the first time in my life that work just didn't feel like work. Most days, I don't even think about it - I just go to a different building and keep having fun. The only time that changed was when the workplace got unhealthy - see the above sections.

It's common to go out drinking/eating at least occasionally with your coworkers. You don't have to drink if you don't want to - many people don't. Some have one drink and are completely smashed. Some will drink you under the table. You can go for a quick bite at a host of restaurants - obviously Japanese, but Italian, French, Burgers, Thai, etc. are all available. You can, contrary to popular belief, have house parties. I've done escape rooms with my coworkers, been invited to theme parks, and been asked to come rock climbing.

A misconception that I feel compelled to address: I have never, ever been excluded because I wasn't Japanese, or invited as a "token foreigner". If you ever get that feeling, you should work with different people because those are just shitty human beings.

My coworkers have, at times, been very enthusiastic to practice their English. You will find that many programmers can read English pretty well. That being said, if you have native or fluent English, you will have a huge advantage in the field because you can read Stack Overflow and error messages with high comprehension. I'm not even kidding - Japanese Stack Overflow is hot garbage, and this will be like a superpower for you assuming you can translate that knowledge back into Japanese if someone asks you the reasoning behind your code.

The company that I'm currently freelancing for has a policy that you should leave when you're done. I have seen people come in at 10 am and leave at 3 pm, after taking an hour-something lunch. There is no stigma, no penalty - just do your job and have fun, and go home when you're done for the day.

I have never had someone criticize me, to my recollection, for taking longer than an hour at lunch. I have been out for two hours at a time, with no issue.

I regularly take 30-minute breaks to think, stretch my legs, etc., and except for a particularly difficult person, have not had an issue, ever.

A serious weakness of mine is that for a large part of my tech career I was chronically tardy. Now I work for companies with no set schedule and remote work (and, ironically, get up at ~7:30 am every day) - but I have never had that be a serious issue for me. Like most things, negotiate it early and be honest. Monolithic companies tend to be less forgiving on this front, however.

Japanese Attitudes Towards People Who Are Not Japanese

Disclaimer: this is a gray area that I have pretty strong feelings about. This is the most opinionated part of this article, and I totally understand if you don't agree with what I'm about to say. It certainly isn't essential reading and is more of a rebuttal to something I see endlessly regurgitated online. If that doesn't interest you, just hop on over to the Overtime section that follows.

I often hear other non-Japanese people say "You will always just be seen as a foreigner". No matter how good your Japanese is or what you know about the culture, no one will make deep friendships with you or you will always be acknowledged as different. I find that often, but not always, this sentiment comes from people who don't have very good Japanese. I want to make something clear.

If you have non-conversational Japanese, that could likely be the case. If we're talking about the government's infamous perspective that people born and raised in Japan aren't Japanese unless a parent was, that is correct (and tremendously shitty for those people at times). If you mean that every time you go to a store, people will assume you don't speak Japanese until proven otherwise / you'll get the English menu because they assume you need one to have a pleasant experience / people will try to practice their English on you as soon as you meet - I am in 100% agreement.

But if you mean that no matter how good you are at Japanese and what kind of people you hang out with, that you'll always be seen as "the foreigner", the issue is more nuanced than that.

I can't speak to other people's experiences, but I live a lot of my personal and work life using Japanese. I learned in Osaka, so I speak like a person from that area, and I can tell you with absolute certainty that people have straight-up forgotten that I wasn't Japanese and I don't look even remotely Japanese. Not even a little.

I've had more than one friend be surprised when we went out and they handed me the English menu, and when I told them it's because I don't look Japanese they laughed and said they forgot.

My friend was asked by a coworker if he was going to vote in Japan's elections. When he said he couldn't, they asked why. He had to remind them that he isn't a Japanese national. He doesn't look "traditionally Japanese", but they assumed he at least had voting rights.

Essentially, if you find yourself in a situation where you feel like you'll never be accepted on the same level because your skin or culture is different, that depends on how you mean it. If you mean that people will assume you aren't from around there at first impression, then of course - we assume a lot about people from the way they look or act.

But if you mean that Japanese people you've gotten to know won't accept you for one of their own, that has not been the case in my experience.

Overtime

Oh boy, the big one.

I've seen it all over the spectrum. I work, on average, about three or four hours of overtime. Per year.

I also personally had someone that called me - in distress - because their company had them working weekends and going home on the last train, every day. This company is known for being particularly abusive during the busy times of the year, and that's certainly not a rare story.

I'm about to follow this with what I think is going to easily be the most controversial, polarizing part of this whole article.

If you work in Osaka or Tokyo, in tech, and you regularly have overtime that makes you unhappy, that is most likely your fault. Hear me out.

Every contract that I've seen in Japan includes a certain amount of "prepaid overtime." This means that every month, for the first x hours of overtime, you don't make anything extra. There are exceptions - significant weekend work, work past 10 pm, etc. can either get you paid more or get you a different day off in lieu. From what I'm aware, even this is a step up from some countries - like the states - where you don't make anything extra at all if you're a salaried worker above a certain pay bracket.

However, if your employer is using this clause to regularly have you work overtime, leave. Japan has an extremely well-developed public transport system, remote opportunities, and a shortage of tech jobs that can land you five interviews per day. I'm not exaggerating - per day. There is no fear of visa renewal for tech companies - it involves them printing off a few pieces of paper about company information, stamping something, and then waiting a few weeks. Many are even willing to sponsor outright. You'll find that there are plenty of companies that will strongly encourage you to leave either on time or early.

I have found that the best way to deal with it is to be really, really upfront. Ask your interviewer what their day is like - what time do they usually come in, what time do they usually leave. If their answer isn't something along the lines of an enthusiastic "I love to leave on time to pursue my hobby of crab-shell collecting", make it very clear that you leave on time as a matter of principle. Actually, make it clear that you leave on time regardless. If that was a dealbreaker, no problem. Go to your other four interviews for that day with places that look cool to work for.

It's a problem in the country - one that's gotten much better in recent years, but a problem just like anywhere else. The difference is, in tech you can leverage enormous demand for your skills to make sure it isn't your problem.

Interviews and Perception of Programmers

Attitudes Towards Programmers

When I tell people that I'm a system engineer ("SE" as it's called here), I usually get the same sort of reaction I imagine programmers get elsewhere. People who have been in the game might talk shop a bit, people who don't get what you do will treat it with reactions ranging from "you must be smart" to a glazed look and a polite nod.

From a corporate perspective, attitudes towards programmers tend to take one of two perspectives. Either programming is like a blue-collar job where you throw people into the code factory for their 10-hour shift, or programmers are a very expensive type of wizard that needs to be enticed with money and benefits to work their magic. Games and sales seemed to think the first, engineering firms and tech-driven products the second.

Interviews, Competition, Company Size

Disclosure - I love interviews. Even when I don't know the answer to a question, I get to learn a ton about something I've never experienced. Plus, if you do know the answer, you're getting paid to be a show-off (assuming you get the job), so revel in one of the few times in your life where that will be socially acceptable.

You will often hear that "You need to work in one place your whole life in Japan." That system has been falling out of use for 30 years but I see it everywhere. Workers have been made replaceable, seen by an increase in contract work. The shift was so intense that it forced the government to make an initiative that if you renew someone's contract for five years, you need to make them a permanent employee. Yes, there are still many companies you can work at for life. You can also change jobs once a year and be fine.

There is very, very little competition because the lack of skilled talent is enormous. If you're just starting out, you'll have an advantage if you have robust projects and fluent English, but if you have actual good domain knowledge in your area? No shortage of jobs. You can get a job here starting at 400k (~4k USD) a month if you put the work in and speak Japanese. If you already have experience and have the chops to demonstrate that, you can pull down significantly more.

I don't have any kind of interview prep advice because I'm sure the way you prep will matter more on your personality than anything else, but when you're in the interview, there are a few things you absolutely should do.

Be honest. Don't mind overtime? Say that. Hate it? Say that too. Salary important for you? Negotiate. People say Japanese people don't negotiate. I can tell you with absolute confidence from being on the hiring side and working with the decision-makers, as well as being the interviewee - when HR gives you an offer, that is not their best offer.

I have negotiated higher salaries, starting bonuses, more vacation days, and different working hours. You can do this - just do it in your style. Don't be afraid to ask over email, compare offers, etc.

Avoid big titan companies (Rakuten, Line) unless salary and reputation beat everything for you because you don't get to negotiate things like flexible job hours and perks. Medium corporations offer a nice balance of salary and perks, startups are often all perks and you have a good chance to negotiate extra things.

In general, if you want something or won't do something, bring that up. For example, I make it very clear in all my interviews that I am a backend specialist, and I don't know frontend - I have basic familiarity with it, I can reason my way around the concepts, but if you asked me to write a webpage the company would grind to a halt.

Most companies do either basic code reviews (fizzbuzz) or some online code test. The online code tests are usually like "Array in an array, print as a grid" type of thing. True story - I was doing a timed code test, screwed up a comma in the output, and didn't have time to fix it before submitting. I submitted it and sent a follow-up email explaining the quick change I would make to have it pass.

At an interview later that week one person said they saw it and had no issue, the other said they didn't even check the test results because they already decided to hire me.

I had a cool interview that was a general whiteboard interview where they asked me to build an architecture for an API. The API was to count fan votes that were cast for their favorite member of a music group during a live broadcast of said group. The catch was that you could only cast a vote if you had a unique code from a cd that you had purchased. That was a blast.

The hardest code review I ever got was at an English-speaking company, I loved it but I didn't know a good amount of the front-end questions.

Line (Chat app and game company) wanted me to take a test to even be able to email their recruiter.

My experiences with Rakuten have been weird - they invited me to a hiring party for mobile app developers, so I went because of free food. It was a fun party but I had two years of experience and none of it was in mobile apps. They wanted three years of app development experience. Weird situation.

Lastly, on the time it takes for closing an offer. I find that the typical turnaround from the first message to offer is about two weeks. Some places are a bit longer, some are a single interview, but that's usually the case. You can expect a casual first interview, at least one follow-up, and then a final offer. It is not uncommon at all to meet the CEO for the final interview - it's often merely a formality, but they frequently sign off on every hire.

Working with Recruiters

I have had the pleasure of working with a handful of good recruiters - just all-around excellent people. I became friends with two of them - one Japanese and one Swedish. I have also had recruiters message me on LinkedIn that have actually read my profile/website.

That being said - much as in America, the vast majority of recruiters are hot trash at their job. Ghosting, mass spam messages, unsolicited friend requests to avoid paying LinkedIn mail fees. It's no better on Japanese platforms, either - mountains of spam that was so bad that I had a template letter (much as I do on LinkedIn) which says "Read my profile, I have x terms". This is often met with silence (LinkedIn) or an apology and an offer to keep in contact if those terms can be met (Japanese platforms).

I have also worked with excellent job sites that I used to find jobs that met my terms. They do exist - just make sure that the site isn't recruiter-oriented. Smaller sites tend to be better, especially ones that follow the format of "Employer posts a job, you can click to show interest."

For people seriously looking for jobs in Japan -
Bizreach is full of spam. Avoid.
Wantedly is usually for companies that have a ton of passion and criminally low salaries that take advantage of new grads. I directly saw this happen. You can find good things there, just be careful.
Gaijinpot programmer jobs seem to have horrible conditions, but I imagine that's because they cater to an audience that might not be expected to have a lot of Japanese skill.

Freelancing

Freelancing is actually surprisingly straightforward. I don't know if you can start as a freelancer without an already existing visa, but you just go to a freelancing company (you 100% need Japanese for this) and tell them you want to freelance. They do the rest of the work helping you find companies and arranging interviews.

Finding work as a freelancer is fast, too. Most companies wanted me to start working as soon as I could.

The primary advantage that I found in freelancing is you can work non-5-day weeks easily and because you don't get paid days off and included insurance you can get a higher salary relatively easily. Lots of freelance jobs have remote work as well.

I work three to four days a week right now, and I love it. I worked a week in the Philippines and have long weekends every weekend.

Jobs That You Can Do Without Japanese

Easing back into meatier topics, there are a few kinds of jobs that you can do without speaking Japanese in Japan, and it comes down to one basic thing - companies that have strong international ties.

This typically means a few choices. If you want a huge salary you should work at a bank, a Japanese branch of a global company, or any similar company owned by foreign interests. If you're interested in passion over money, there are English-only startups that sponsor visas- they are few and far between, but they exist. If you want something in the middle, Fintech is a good middle balance - decent salary, good passion.

All of the above require you to be good at what you do - if you don't have sufficient experience, it's very unlikely that you'd be hired over an inexperienced-but-English-proficient local.

Money, Visa, Location

Salary and Cost Of Living

I want to get a frame of reference before we start talking about salary - and we will, in very concrete detail, with numbers. There is an opinion that in general, Japan pays programmers poorly and Tokyo is crazy-expensive.

We need to tackle these things one by one, and I'm going to start with the "Tokyo is crazy-expensive" one first.

You can live in a great apartment for very cheap (well less than 100k yen/$1k per month) if you're willing to commute an hour or more on trains. I mean like shiny, attractive apartments with a ton of space. Houses with parking spaces. You can also pay easily more than that for a single room about 10 minutes from Shibuya (a main hub of Tokyo).

Japanese apartments are undeniably smaller than their western counterparts. Tokyo is no exception, and neither was Osaka. That being said, in Osaka I lived in a one-bedroom, living room, dining/kitchen for 60k (~$600)/month. In the greater Tokyo area, I've lived an hour from work in a two-bedroom, dining-kitchen with a living room for about 90k (~$900) and a one-bedroom dining-kitchen with a living room in a more expensive, nicer area of Tokyo for 120k (~$1,200).

Right now, I live in an apartment that was built the month before I moved in. It's 30 minutes from Shibuya (4-minute walk to my station). 10 minutes to a different close-by main station. One bedroom, separate sunlit study, combo living-dining room, and a bar-kitchen that runs 155k (~$1,550)/month. From what I've gathered that is insanely cheap compared to NYC or the Bay Area. That being said, those are still big numbers depending on your fiscal status.

I move next month (working remote means I can live farther now for cheaper) to a slightly bigger place with the same layout and parking for about half the cost.

As far as where to live - Osaka for the people and culture, Tokyo for the pay and job opportunities. I was very happy in Osaka, but I hit a pay ceiling after my first job. Tokyo has a massive amount of work and good pay to boot.

Now, salary and how Japan pays programmers.

I'm going to list all of my jobs and the salary that I made. This doesn't include personal jobs, my side business, etc.

Starting - Games. Osaka. 280k(~$2.8k)/mo. In Osaka as a single male, this was enough to have an okay standard of living.
Next year - Outsourcing Shop. Yokohama (near Tokyo) - 400k (~$4k)/mo. This was pretty comfortable, but not great.
Next year - Engineer at a sales and recruiting place. Tokyo. 708k (~$7k)/mo, with 300k (~$3k) signing bonus to cover moving costs. A comfortable lifestyle.
Now - Freelancer. My current rate depends on the job, but I charge about 750k (~$7.5k)/mo, some jobs could cost more, rarely less. Comfortable lifestyle. Note that this is technically less because of the way that taxed and health insurance work here.

At each job, I went home and studied/built things that I wasn't doing at work. As soon as I had studied the essentials of whatever tech we were using at the office, I would chase some side project or build something for money. As a result, my salary may be higher than "normal" because of my deeper skillset.

At each step, I negotiated. From job one. If you don't mind working in more traditional "10-7 C++ or C#" types of jobs, there are plenty that break 10m ($100k)/yr.

I found that job offers start to thin around 8m ($80k) a year. This is where you'll have fewer interviews because fewer companies can afford you - this is a sign you're rising above the market average.

There is this commonly touted number online that engineers make 100K+ right out of graduating. What is less touted is the number of those that still need a roommate to live close to work.

Seattle, NY, Bay-area, you can expect to make a lot of money, and pay a lot in rent. Tokyo is not a cheap place to live, but the only real difference is apartment size. By the square foot, Tokyo is very expensive. Apartments tend to be smaller, but I can live without roommates comfortably. I have tons of space and live in a relatively expensive area of Tokyo - other similar places would have set me back much less but be further out. There is no Tokyo hustle-and-bustle in my area - no train sounds, no traffic noise. Aside from me, my area is super quiet.

Apartments

Ugh. I have had such a mixed bag with this. Hunting for them has been JAM PACKED with racism at times, and incredible easy at others. Finding a job is fairly trivial, finding a place to live was insane.

I am in no way exaggerating when I say that some of the moving experiences I had in Tokyo and Sendai were the most discriminated against I have ever been in my life. When house hunting once, out of the exactly 30 apartments that I wanted to see in Tokyo, only 3 would even let me look. The rest straight-up said non-Japanese not welcome. Japanese skill, salary, relationship status - none of it mattered. The apartment I got I liked, though.

Outside of that specific time, every other time I've done this in either Tokyo or Osaka it was really easy and super cool. The real estate agents were helpful (if your real estate agent isn't aggressively finding places for you and calling them, go across the street to another one). Every apartment was super cool about having me move in except for one oddball that wanted me to be double-insured because I wasn't Japanese. I laughed and said no. In general, they pretty much asked: "Does he speak Japanese?" They didn't ask about my salary or anything. I think a few asked how long I'd been/be in Japan, and if I had some kind of job already, but that was it. My Osaka one wanted to know how long my visa was good for - and threw in an air conditioner for free because they thought I was a fun person.

Unfortunately, my next move to the countryside involved all of my candidate homes allowing me to come to take a look, but a Japanese person would have to be my guarantor or I couldn't move in (be double insured). They made it very clear that it was because I was not Japanese. I've heard that the reason may be that it's the countryside where there are fewer non-Japanese, and I'm inclined to believe that reason.

Conclusion

Just like every other country, it's a mixed bag. You'll have companies that you dislike strongly, you'll have companies that you love.

You'll have people that frustrate you endlessly, and people that will leave positive lasting impressions on you for the rest of your life.

You'll feel underpaid, you'll feel overpaid. You have expensive apartments in the heart of the city, and cheap ones on the edge.

You'll make well more than the average earner in Tokyo, and have a comfortable life if you can speak Japanese or have intense skill.

Life here is really good - for me. Your experience might be different, but from what I've seen it's really the same here as it is everywhere else.

It's what you make of it.

If you have questions or comments about living in Japan, please leave a comment and I'll do my best to get back to you.
Also, I now have a Twitter! My DMs are open, so shoot me a message if you need anything.

How to use REST clients / native apps to make Shrine client uploads and S3 work for you (no Javascript, just backend)

Rob Sherling — Tue, 26 Feb 2019 04:52:18 +0000

Shrine and AWS

I wanted a direct upload (client uploads a file, and then passes a link to the server) solution for a new app that I'm writing. The backend was going to be shared amongst a lot of different services. I looked around - Shrine seemed the most promising, so I went for it.

A lot of this info just applies to S3 direct uploads and API signing in general, so if that confuses you, read it anyway.

I didn't want to use a front end solution with a premade library, because I was trying to figure this out for a mobile app. I think I did - if you can do it in REST, you can do it anywhere.

I'm writing this as I have just figured out how to make Postman/Insomnia work, and I'm not gonna lie - I'm pretty annoyed. It really should not have taken days to get an easy example of how to use Shrine without subscribing to GoRails (I did) or using the Uppy library (Not an option a lot of the time).

Quick aside - While GoRails really wasn't worth it for me and I cancelled almost immediately when they used JS as well, Chris (one of the tutorial guys) was super cool and easy to understand in his videos. Not knocking the service as a whole, they just didn't have what I needed. Well, except for the sweet, sweet test image I used (Google "Money Sloth").

Okay. On to the main part.

The Set Up - The Break Down

Setting up Shrine is actually pretty straight forward for the server side. I'm going to skip the entire thing because the tutorials are really straight forward, I just want to explain the AWS part and give you a quick tip. Make sure to set up your bucket with CORS like shrine tells you, except insofar as we are concerned, leave the accepted domains to * because we're rest-client testing.

Quick tip - if you're hesitant to use shrine because you want to use devise or some kind of auth solution to protect your presign endpoints (and you should!), rest easy. The docs are straightforward.

We use POST over PUT because of a buncha reasons that are best articulated here (read the link, read the answer).

Quick overview of how it all works

You want the client to upload images to S3 without your server having to handle them. However, if you just let anyone upload anything, you're going to have a terrible day some day when you find out that someone has been using your AWS account to store their exotic collection of Giraffe Dancing videos. All 20 terabytes of it.

Signed posts let you avoid that problem by doing the following, in order -

1) Be authenticated and ask your server for permission to upload a file. You tell them what type of file (you CAN tell them the filename, but no. No need). For example, "I will upload a jpeg."

2) Shrine runs its magic and generates a BUNCH of specific stuff to let you do that. Like, and I am serious, 8 things that you will need to upload the file in addition to the file itself.

3) You take that mountain of information and do the upload.

ERROR LIST

If you see one of these errors, here is how you fix it:

Some error related to expired request - Expiration time has passed, do it again.
Some error related to 403 expired request policy something - your server time is messed up. In my ABSURD case, I had updated Postgresql through homebrew which screwed up the database clock. I restarted. Solved.

Some error related to signing - One of your headers is messed up. Of note - some policies, when generated, have an = sign at the end of them. Make sure that is written / deleted correctly when copy pasting.

Some error that says "Policy something something "eq", "Some header", "some value"" - you're missing that value. Put that value IN THE FORM DATA with the key/value that it says in the error. i.e. "You are missing eq, content-type, image/jpeg go into your form data and add key:content-type, value:image/jpeg

Before we even start

The post request that you send to S3 must be form-data. You don't need any headers, at all, because the form data will have everything you need. It must, obviously, be a post request. This next part is super, super, super important: the key for the object to be stored in the bucket MUST COME FIRST, and the attached file MUST COME LAST. Your post will not work if that doesn't happen. Also, once you write the whole request, if you aren't getting the file to post go ahead and check the actual curl code behind the request. Look for the part that says "filename=" and make sure the filename is actually attached. Mine didn't until I upgraded to the latest version of postman.

Now we can start

Okay. Wow. Let's actually start this.

First send a simple get request to your presign endpoint on your server. I set mine to http://localhost:3000/api/s3/params.

The get request should have one query parameter - "type: "

For example, if your rails code looks like this in your presign endpoint(and it should, more or less):

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: 'cache', **s3_options),
  store: Shrine::Storage::S3.new(**s3_options)
}

Shrine.plugin :activerecord
Shrine.plugin :restore_cached_data # refresh metadata when attaching the cached file
Shrine.plugin :determine_mime_type
Shrine.plugin :presign_endpoint, presign_options: lambda { |request|
  filename = request.params['filename']
  type     = request.params['type']
  {
    content_disposition: ContentDisposition.inline(filename), # set download filename
    content_type: type, # set content type (required if using DigitalOcean Spaces)
    content_length_range: 0..(1 * 1024 * 1024) # limit upload size to 1 MB
  }
}

At the lines

filename = request.params['filename']
type     = request.params['type']

That means that our GET request should use type as the key. If you set it to something like type = request.params['content'] then the key name should be content. Regardless, a valid param might look like type=image/jpeg(url escaped by your REST client, of course)

We actually don't need the filename param. You can include it, but if the filename param doesn't match exactly it won't send. So if you have a picture of say, a gorgeous money sloth and you want to name it something like MONAY.jpg, if you send the filename as MONAY.jpg and then attach that from your rest client, you won't get it to work. This is because your rest client will attach it as /User/me/Documents/MONAY.jpg, and those names do not match when you upload.

Anyway.

Okay, so after you send your straightforward get request (make sure to include auth headers if you're using an auth solution), you'll get back something that looks like this:

{
  "fields": {
    "key": "cache/ac48d4a7764610579df3f26c316f3947",
    "Content-Disposition": "inline",
    "Content-Type": "image/jpeg",
    "policy": "eyJleHBpcmF0aW9uIjoiMjAxOS0wMi0yMlQwOTo1MjozMloiLCJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJ2aXNpb24td2ViLWltYWdlLWRldiJ9LHsia2V5IjoiY2FjaGUvYWM0OGQ0YTc3NjQ2MTA1NzlkZjNmMjZjMzE2ZjM5NDcifSx7IkNvbnRlbnQtRGlzcG9zaXRpb24iOiJpbmxpbmUifSx7IkNvbnRlbnQtVHlwZSI6ImltYWdlL2pwZWcifSxbImNvbnRlbnQtbGVuZ3RoLXJhbmdlIiwwLDEwNDg1NzZdLHsieC1hbXotY3JlZGVudGlhbCI6IkFLSUFKT0VSVTNMTkFaSExOUFRBLzIwMTkwMjIyL2FwLW5vcnRoZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotZGF0ZSI6IjIwMTkwMjIyVDA4NTIzMloifV19",
    "x-amz-credential": "********/20190222/ap-northeast-1/s3/aws4_request",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-date": "20190222T085232Z",
    "x-amz-signature": "769489a916a228d142e9bff18688bf45f44c7e77eac81f4797b8c48e3b8ab756"
  },
  "headers": {},
  "method": "post",
  "url": "https://BUCKETNAME.s3.ap-northeast-1.amazonaws.com"
}

Don't panic. One by one -

key - what shrine named the object in s3 for you.
content-disposition - Irrelevant for us, but docs anyway. Just copy it.

Content-Type - yep, this is what we said our type was.
Policy - just copy it. This tells Amazon what is permitted to be uploaded, and how.

The next four that start with x-amx-* - these four, together, are part of the Amazon V4 signature protocol. They tell Amazon that the request is legitimate. That's really it, just make sure to copy them correctly every time and you won't have issues.

Headers - this is messed up. Ignore these entirely. I spent way, way too long messing with headers before I realized they were red herrings.

Method - Post it.
Url - Post it to this Url.

Okay, now I'm just going to go ahead and show you what your client needs to look like.

If you are using postman, copy your URL into the url bar and set it to post. Click the body tab, and set it to form-data. Now click the bulk-edit orange text in the upper-right hand corner, and past this in -

key:YourKeyHere
x-amz-credential:YourCredentialsHere
policy:YourPolicyHere
x-amz-algorithm:AWS4-HMAC-SHA256
x-amz-date:YourDateHere
x-amz-signature:YourSigHere
Content-Disposition:inline
Content-Type:YourContentTypeHere

Just replace the values with your values, add the file at the bottomand you're golden. Or, if you are using another rest client, use the above keys and fill in your own values a bit slower because you won't have bulk copy-paste, so you need to go line by line. then add the file at the bottom

Okay. Now, hit send. If everything worked, you should get back a 204, and the image is in your bucket. If everything did NOT work, weep and look at the error list above, or post a comment and I'll do my best to troubleshoot on the roughly one day/week that I read comments.

Using that response to save the file

This one is actually pretty simple. Make sure that you have the :restore_cached_data plugin available (see my Shrine code, above). In your Ruby code, once you've authenticated and such, call the following:

ModelInQuestion.update_attributes(attachment_name: {id: "ID OF S3 OBJECT", storage: "cache"}.to_json)

For example, if your object is something like books with your image uploader that you set up in the folder set to image:, assuming your AWS response looks like this:

x-amz-id-2 → stuff
x-amz-request-id → stuff
Date →Sat, 23 Feb 2019 10:04:20 GMT
ETag →"daa14e86f803fca11a1baff256fa259b"
Location →https://bucket.s3.ap-northeast-1.amazonaws.com/cache%2Fdabe031bee03ae8365e2ec48b523bb33
Content-Length →0
Server →AmazonS3

You want to call this code:

YourBookModelHere.update_attributes(image: {id: "dabe031bee03ae8365e2ec48b523bb33", storage: "cache"}.to_json)

We get that image ID from the end of the location tag. The documentation actually has a regex that parses that for you - last paragraph

Quick note - I found that if you want to pass the string in and use the regex, you're going to have to take the url string, restore it, and then remove the beginning of the line character from the regex.

Or, in code:

URI.decode(params[:link])[/cache\/(.+)/, 1]

And now when you run that update attributes, you should get this cool thing back:

<ImageUploader::UploadedFile:0x00007fbb995c8eb8
@data={
"id"=>"e68f1910d4fb723d6538383dbbac5f78",
"storage"=>"store",
"metadata"=>{"filename"=>nil, "size"=>92931, "mime_type"=>"image/jpeg"}}>

note that if your mime_type is nil, you need to add Shrine.plugin :determine_mime_type to your plugin list.

Now, if you call yourbook.image_url, you get a super-cool signed url that lasts for 15 minutes!

Lastly, if you want to set your expiration time for something a bit more narrow than 15 minutes (say, 5 minutes), use the default_url_options plugin -

  # 90 seconds to use the upload URL, 5 minutes to use the GET url
  Shrine.plugin :default_url_options,
    cache: { expires_in: 90 },
    store: { expires_in: 300 }

Huge shout out to Janko the gem maintainer for all of his help in the Shrine Google group. I recommend going over there if you have questions - he's very friendly!

How to REALLY set up a React Native environment on Mac

Rob Sherling — Thu, 09 Aug 2018 11:39:10 +0000

This is basically taken from a private README that I use and cleaned up for public consumption. It doesn't really have the humor that I try to bring to these things, and I tried to keep it simple for Japanese speakers that I can point here when they need help. Very quick and dirty, may clean up later.

This is only for Mac. If you aren't using Mac for React Native dev - do it, it's pretty great.

Setting up a react native environment

Steps

Software installs and file setup

Install Xcode from the app store - Xcode is a program that runs iOS apps for you, and lets you start a simulator for your app so you can develop without needing a physical iPhone.

Install a Java Development Kit - you need this to make Android apps in React Native
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Accept license agreement, Mac OS X x64

Install Android Studio (google this). This is how we will run android simulators so we don't need an Android phone open to check our work. Now, we need to set it up.

Open Android Studio. Choose custom install. Check the boxes for

Android SDK
Android SDK Platform
Performance(Intel@HAXM) - this speeds up the simulator.
Android Virtual Device

Begin downloads.

Install Node and Node Package Manager(NPM). If you don't know what that is, literally google - install Node on MacOS. It's very painless

That will take a long time, so in your terminal, enter the following (without the $ symbol).

This will install watchman, a program that watches files and updates your app when they change.
$brew install node watchman

This will install react native so we can do cool stuff like make apps
npm install -g react-native-cli

Back to Xcode

install the xcode command line tools. In terminal:

$xcode-select --install

Add the following line to your ~./bash_profile

alias android-studio="open -a /Applications/Android\ Studio.app/"

What this does (after restarting terminal) is let you open directories by typing

android-studio .

in that directory.

Android SDK and AVD Install

Open android studio once downloads finish, click on "configure", go to SDK manager.

Click "Show package details"

Enable Android 6.0 (Marshmallow), and INSIDE Marshmallow, enable

Google APIs
Android SDK platform 23
Intel x86 Atom_64 System Image
Google APIs Intel x86 Atom_64 System Image

Do the above for Android 7 and 8 as well (Not necessary, but good to have for the future)

Next, on SDK tools tab, click "Show package details", look for "Android SDK Build-Tools", and make sure that 23.0.1 is selected
click apply.

append to ~/.bash_profile:

export ANDROID_HOME=$HOME/Library/Android/sdk
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/platform-tools

terminal: source $HOME/.bash_profile

Confirm sdk location with echo $PATH

We just did all that because really we have no choice. That's the work necessary to make Android apps.

New project

Setup a project really quick to configure the emulators.

In terminal, start a new project with
$npx react-native init <APP_NAME>

$cd <APP_NAME>

$npm install

Note, you can use Yarn as well (google install Yarn)

Running on iOS

In a terminal, type $react-native run-ios from your react native project folder

Confirm that app runs.

SIMULATOR NOT FOUND ERROR

If it tells you that a simulator isn't found, restart your computer. Common issue on using react-native init command.

Press command+d

Enable hot reload.

Change text in app.js under ios, save, and observe instant change in simulator.

Close simulator.

Celebrate.

Running on Android

Open the project in Android Studio. (Go to the react-native directory, go to /android, then run android-studio .

Open Android studio, and open the Example App directory in it
When it says "Android framework detected", configure -> Agree

click the AVD manager icon in the upper-right hand corner (Waaaaay upper right)

Grab whatever device you want to test on, then click Next
on the x86 images tab, then look for the Marshmallow API Level 23, x86_64 API image with a Android 6.0 (Google APIs) target.

Click next, then finish.

Click the play button on your specific android virtual device to launch it. When the home screen appears, you're ready!

terminal in project root:

react-native run-android

Press cmd + m

Enable hot reloading

IF YOUR SCREEN IS WHITE:

Go to more settings (sidebar), go to settings, click advanced, then set the OpenGL ES renderer to a new option (randomly), then exit the settings, exit the emulator, restart the emulator, and re-run react-native run-android

Change text in app.js

Observe change.

Celebrate!

Editor

I was hardcore Nuclide (Atom). Now I'm all about that visual studio code.

Setting Up ESLint

This Project uses ESLint.

Source:
https://github.com/expo/eslint-config-universe
https://github.com/Intellicode/eslint-plugin-react-native

run the following in the root directory:

yarn add --dev eslint prettier eslint-config-universe eslint-plugin-react eslint-plugin-react-native

Move this file to your project root https://gist.github.com/Rob117/118443a2610af3abcbb3d8ddb8a213d1

in your package.json, add the following line under "scripts"
"lint": "./node_modules/.bin/eslint --fix *.js **/*.js"

Then just run npm run lint to lint your code.

Installing testing tools

Jest is already installed

To run jest tests: npm test -- __tests__

I realize that this is very rapidly written and might be a bit hard to follow - if any part doesn't work, let me know and I'll update it. I wrote this in a time crunch as a promise to a friend.

Circle CI Nightly Builds and Github PRs with AWS Lambda Triggers

Rob Sherling — Mon, 23 Apr 2018 14:09:18 +0000

Hi! In this blog post, I'll show you in detail how to create daily/nightly builds in CircleCI with AWS Lambda, and take the results of those builds and create a Github pull request with the changes. Everything in this tutorial is 100% free.

Repository Here

For this project, I am assuming that you already have your project on CircleCI and Github. If you don't use Github, the Lambda and CircleCI parts will still be useful to you, just ignore the part at the very end where we create pull requests. If you don't have CircleCI, this project unfortunately has almost nothing for you unless you want to see an example of what a Github API request looks like.

CAVEAT: In the bash script, we use the line git add apples.txt to add our newest files. I would recommend writing git add <whatever your files are expected to be> instead of some catch all expression like git add .. For example, if you ran bundle update, I would recommend running only git add Gemfile.lock. This is because CircleCI can mess with your repo in crazy ways when it sets up for things. Example: Before it gets to your code, it may run something like bundle install to get all of your project dependencies. This means that you will suddenly have tons of files related to gems in vendor/bundle, and if you didn't gitignore that file, you are looking at a huge diff. It can also edit database.yml, so there's that.

Along the way, I'll be explaining gotchas or odd lines of code. Let's begin.

The way that nightly builds work is that first, we'll make a bash script to run all the code we want to execute on the build machine (in this case, make a file). This could be anything from rails t to bundle update to a bash script that checks the weather around your Singapore server (if you write that, please show me, I would be super interested). The end of that script will have us push any changes we made to Github as a pull request.

Then, we'll put that script in our CircleCI File to make it activate when we trigger our builds via API. This is relatively simple, a few lines.

Finally, we'll build a lambda and slap a cron job on there to make it proc our build at whatever time and frequency you please. Note: Lambda cron is limited to a rate of no greater than 1/minute (source).

First, the bash script. Create a file with a name that describes what the code will do. I wrote a script called bundle_update.sh, because we're gonna be updating bundles. Don't forget to chmod +x !

bundle_update.sh


# First check out the branch you want to work code magic on. This is our base.
git checkout 'master'
# Force our branch to be in alignment with remote.
git reset --hard origin/master
# Make sure the commit is made with the correct credentials
git config --global user.email "<Your email here>"
git config --global user.name "circle-ci"
# Check to see if our work branch exists - $? command will return 0 if branch exists
# of course, this branch name could be anything you like, it's just important that it is unique to this script.
git rev-parse --verify circle_ci/make_apples
# If we have no branch, make one. Else, switch to it.
if ! [ 0 == $? ]; then
  git checkout -b 'circle_ci/make_apples'
else
  git checkout 'circle_ci/make_apples'
fi

# Run your custom code here. We'll just make a file for ease of use.
# This code can be anything you want to do, bundle update, nginx ip configs, etc.
touch apples.txt

# If we have changes, make a PR for those changes
if [[ `git status --porcelain` ]]; then
  # Ensure that our branch is correct. The last thing you want is to force push into the wrong branch
  if [ `git rev-parse --abbrev-ref HEAD` == 'circle_ci/make_apples' ]; then
    git add 'apples.txt'
    git commit -m 'My commit message here'
    # -f Necessary to possibly overwrite any existing remote branch code and force this to be correct
    # -u sets upstream for circleci
    git push -f -u origin circle_ci/make_apples
    # Head = Code to merge, base = branch to merge into
    # $GITHUB_ACCESS_TOKEN is an environment variable we set on CircleCI, but you must set it on your local machine if you want to test there as well
    # Example Github URL: https://api.github.com/repos/Rob117/circletest/pulls?access_token=$GITHUB_ACCESS_TOKEN
    curl -H "Content-Type: application/json" -X POST -d '{"title":"Title Of Pull Request Here","body":"Body of pull request here", "head": "circle_ci/make_apples", "base":"master"}' https://api.github.com/repos/<Organization or User Name>/<Project Name>/pulls?access_token=$GITHUB_ACCESS_TOKEN
  fi
fi

Before I explain the gotchas of the above code, go onto Github and create a token. That token should have the repo checkbox ticked so it has Full control of private repositories. Save that token. Switch over to CircleCI, and on your project settings page under Environment Variables, create a variable with the name GITHUB_ACCESS_TOKEN and a value of whatever your token string was.

Also ensure that your project has ssh permissions for Github set as well, or we can't push code from within our scripts. You can check by going to Checkout SSH Keys in the project settings and ensuring that you have a key that is permitted to ssh to your git repo there. If not (i.e., you don't have a user key there), click add user key and authorize the application to use your key. After authorizing, click on the 'create and add user key' button to get your ssh key on the machines so they can push code.

If you want to test the code on your local machine, go ahead and export GITHUB_ACCESS_TOKEN=<string here>.

Okay, now that we can actually run this code, let's break it down.

Most of the comments explain the code clearly. Remember to replace your email in the git config section, and your Github API URL organziation name and project name in the last line. The gotchas are:

We have to reset our base branch to whatever it is on remote with git reset --hard origin/develop. The reason for this is because as a byproduct of using CircleCI, a lot of things happen to our build machine repository before we execute a single line of code, and they could taint the final results. Syncing develop with its remote counterpart ensures that we have a pristine copy, so it's just good practice anyway.

These two lines:

git rev-parse --verify circle_ci/make_apples
if ! [ 0 == $? ]; then

The first line checks to see if the git branch exists. If it does, it will show 0 when you run the command $?, else it will show something else. This tells us if the branch exists so we know whether to make one or not.

if [[ `git status --porcelain` ]];

Simply runs git status and outputs nothing if there is no change. If this outputs text, we want to make a pull request.

if [ `git rev-parse --abbrev-ref HEAD` == 'circle_ci/make_apples' ]; then

Checks if our branch was actually created correctly. We do this so that we are sure we aren't going to force push to the wrong branch.

curl -H "Content-Type: application/json" -X POST -d '{"title":"Title Of Pull Request Here","body":"Body of pull request here", "head": "circle_ci/make_apples", "base":"master"}' https://api.github.com/repos/Rob117/circletest/pulls?access_token=$GITHUB_ACCESS_TOKEN

This is the magic. Base is the branch we started with in the beginning, the one that we want to reflect our code changes post-merge. Head is the branch we created in this script to do all of our work. We take the changes and make a pull request using our github access token that is stored in our environment variable.

Okay! We've broken down the bash script. Remember- just change the name of the base branch, created branch, and the file creation part to use the script in your own project. Running this script on your local environment - assuming you set your environment GITHUB_ACCESS_TOKEN var - should work smoothly and create a pull request on github.

Next, let's edit our circle CI. It's just a few lines of code added to your normal circle.yml:

# Other Circle.yml code

test:
  post:
    - >
      if [ -n "${RUN_NIGHTLY_BUILD}" ]; then
        # If you put the script in your scripts folder, the path is ./scripts/<name>
        ./bundle_update.sh
        # As many scripts as you want here
      fi

# More circle CI code

This code says 'if we get a post request that has variable called RUN_NIGHTLY_BUILD, execute the following scripts'.

That's it! Push this code to the branch that you want to call it from. For example, maybe you want to keep all of your scripts in your builds branch. Simply merge the above script and yml file into a branch called builds, and push that branch. You're all set to trigger your builds! You could also use a staging branch, develop branch, etc.

SUPER IMPORTANT NOTE: If you have more than one script set to run and you only store the scripts in a single branch - say, builds - and you change branches within a script, you MUST run git checkout builds between script runs / at the end of each script. If you don't, Circle CI may fail to run all your scripts because you'll have changed to a branch where they don't exist!

Okay, next let's set up our lambda and be done with this.

First, hop on over to Circle CI, go to project settings, then go to API Permissions, then click Create Token. The scope of the token should be All, and the label can be whatever you want. I chose Lambda Execution.

Save that token, and open up your AWS Control Panel. Switch over to Lambda.

Click Create Lambda Function, then click Blank Function. Next, click the empty dotted box to the left of the word Lambda and click Cloudwatch Events as the trigger. For the rule, create new rule. Rule name is Nightly-Build and the schedule expression is cron(0 0 * * ? *) -run at 12 UTC every day. (Source)

Check Enable Trigger, then move to the next screen.

Name the function 'NightlyBuild'. Skip the code for now.

Set the environment variable CircleCIToken to .

Set the role to a basic lambda execution role.

Under advanced settings, set the timeout to 15 seconds.

Finally, the lambda code:

// Based on: https://github.com/godfreyhobbs/circleci-cron-serverless/blob/master/handler.js
'use strict';
var http = require("https");

exports.handler = (event, context, callback) => {
    var options = {
        "method": "POST",
        "hostname": "circleci.com",
        "port": null,
        // example: "/api/v1.1/project/github/RobUserName117/secret-project/tree/builds"
        "path": "/api/v1.1/project/<github or bitbucket>/<organization or user name>/<project name>/tree/<branch name with scripts>" +
        "?circle-token=" + process.env.CircleCIToken,
        "headers": {
            "content-type": "application/json",
            "cache-control": "no-cache"
        }
    };

    // Make request, write out all of response
    var req = http.request(options, function (res) {
        var chunks = [];

        res.on("data", function (chunk) {
            chunks.push(chunk);
        });

        res.on("end", function () {
            var body = Buffer.concat(chunks);
            const response = body.toString();
            console.log(response);
            callback(null, response);
        });
    });

    req.write(JSON.stringify({
      "build_parameters": {
        "RUN_NIGHTLY_BUILD": "true"
      }
    }));
    req.end();
};

Most of this code is self explanatory. The line that I would like to direct your attention to is the "path" part of your options object. According to the Circle CI docs, the URL that you post to is /api/v1.1/project/vcs system/organization or user name/project name/tree/branch

So if you wanted to push to the staging branch of the blogly project of the Google organization, stored in github, you would use:
/api/v1.1/project/github/google/blogly/tree/staging

The second thing is that we are calling our builds by sending a JSON with the RUN_NIGHTLY_BUILD parameter set to true. You could set or send as many variables as you please - see the docs for more details.

When this code runs successfully (click 'TEST'), you should get a response that says something like "Last build was blah blah blah". If you see a response that looks like CircleCI Webpage HTML, you made a configuration error somewhere.

Switch over to CircleCI and watch your build happen, then switch to Github and witness your glorious pull request!

Finished! Thanks for reading to the end. If you have any questions, please message me directly or leave a comment.

Enhanced Active Admin - Audio and Image Previews

Rob Sherling — Sun, 04 Jun 2017 08:37:29 +0000

Images and Audio Preview in Active Admin

This is a small piece of code that extends active admin to show previews of images and audio on the index, edit, and show pages. It's very straightforward and very, very easy to implement.

repo: https://github.com/Rob117/active-admin-uploads

Why

I was recently working on a project where I needed a portable solution for enabling image and audio previews in active admin. The admins needed to be able to click on any given piece of audio to hear it, or an image preview to see it full size. They also needed this information available in the show, index, and edit views. This is the code that I distilled to perform that function. The full walkthrough is at the bottom.

Note: This code is not very Safari friendly, so I strongly recommend Chrome or Firefox for working with active admin (and also in general).

The Code

Module EnhancedUploader
# We grab the form, then the object from the form. We try to then show the
# content of that object
  def input_audio(f, property)
    f.input property, hint: audio_content(f.object, property)
  end

  def input_image(f, property)
    f.input property, hint: image_content(f.object, property)
  end

# We simply grab an object and check to see if our property exists on it.
# If it does, display the object
  def audio_content(object, property)
    if object.persisted? && object.respond_to?(property)
      audio_tag object.send(property), controls: true
    else
      content_tag(:span, 'No Current Audio')
    end
  end

  def image_content(object, property)
    if object.persisted? && object.respond_to?(property)
      image_tag object.send(property), height: 150
    else
      content_tag(:span, 'No Current Image')
    end
  end
end

Inputs Explanation

First, we'll start with the input form options

  def input_audio(f, property)
    f.input property, hint: audio_content(f.object, property)
  end

  def input_image(f, property)
    f.input property, hint: image_content(f.object, property)
  end

So, in order to understand what this code does, you have to understand what a hint is in active admin. When you edit something in active admin, it allows you to give some context as to what that object might be. You typically specify that with the hint: param. For example, you could write some text next to a field called 'credit card' on an input form that says 'application only accepts visa.', and it would show that next to the input field.

Let us assume we have a user profile picture that we want to display a preview of when we edit it on a form. We also have a audio sample of the user saying 'hello' that we would like to be playable on the form. The code for the input image and audio would look something like this:

  # f is a form
  form do |f|
    extend EnhancedUploader
    input_image f, :profile_picture
    input_audio f, :hello_audio
  end

Please check the section "The Use" for a minimal, guided, working example.

Audio and Image Content methods

Next, we'll explain what the following audio and image rendering code does:

  def audio_content(object, property)
    if object.persisted? && object.respond_to?(property)
      audio_tag object.send(property), controls: true
    else
      content_tag(:span, 'No Current Audio')
    end
  end

  def image_content(object, property)
    if object.persisted? && object.respond_to?(property)
      image_tag object.send(property), height: 150
    else
      content_tag(:span, 'No Current Image')
    end
  end

This code is much easier to explain. In each method, we accept an object. First, we check that it is persisted (saved to the DB), because if it isn't there's nothing to show or render - we cannot render data that we do not have.

Next, we check to make sure that the object can actually respond to the method that we want. It wouldn't be a good idea to try to render something that the object doesn't have! For example, before we try to display a profile_photo, we should make sure that the model actually has a profile_photo field that we can show.

Lastly, we simply render an html5-compliant image link or audio player with controls. The code in the show section would look something like this:

  show do
    extend EnhancedUploader
    attributes_table do
      row :profile_photo do |row_object|
        image_content row_object, :profile_photo
      end
      row :hello_audio do |row_object|
        audio_content row_object, :hello_audio
      end
    end
  end

Walkthrough

Laying Foundation

rails new demo
rails g model User name:string location:string
rails db:create db:migrate

Now we add active admin and devise
In your gemfile, add the lines

gem 'activeadmin'
gem 'devise'

bundle
rails generate devise:install
rails g active_admin:install
rails db:migrate db:seed
rails s

Go to localhost:3000/admin, login with
username: admin@example.com
password: password

create file at app/admin/user.rb, and paste in this content:

ActiveAdmin.register User do
  permit_params :name, :location
  # Show the essential data in the index
  index do
    selectable_column # we can select columns for batch actions
    column :id
    column :name
    column :location
    column :created_at
    column :updated_at
    actions
  end

  # When you click on the show for any individual item, this data is rendered
  show do
    attributes_table do # display the following attributes
      row :id
      row :name
      row :location
      row :created_at
      row :updated_at
    end
  end
  # When you click on edit, this form is rendered
  form do |f|
    f.semantic_errors
    f.inputs do
      f.input :name
      f.input :location
    end
    f.actions
  end
end

Go to http://localhost:3000/admin/users and confirm that we can create, delete, and edit our basic users.

Adding CarrierWave

Time to add an image uploader and an audio uploader. First, let's include our library that will manage data display for us. Move the following file to lib/enhanced_uploader

+git file+

Add the following to your gemfile:

gem 'carrierwave', '~> 1.0'

bundle

make two uploaders, one for profile_photo and one for hello_audio
rails g uploader profile_photo
rails g uploader hello_audio

Add both uploaders to the user model

rails g migration add_profile_photo_to_users profile_photo:string
rails g migration add_hello_audio_to_users hello_audio:string
rails db:migrate

Open app/models/user.rb file and mount uploaders

class User < ApplicationRecord
  mount_uploader :profile_photo, ProfilePhotoUploader
  mount_uploader :hello_audio,   HelloAudioUploader
end

Replace app/admin/user.rb with the following (we aren't showing the enchanced uploader quite yet):

ActiveAdmin.register User do
  permit_params :name, :location, :profile_photo, :hello_audio
  # Show the essential data in the index
  index do
    selectable_column
    column :id
    column :name
    column :location
    column :profile_photo
    column :hello_audio
    column :created_at
    column :updated_at
    actions
  end

  # When you click on the show for any individual item, this data is rendered
  show do
    attributes_table do
      row :id
      row :name
      row :location
      row :profile_photo
      row :hello_audio
      row :created_at
      row :updated_at
    end
  end
  # When you click on edit, this form is rendered
  form do |f|
    f.semantic_errors
    f.inputs do
      f.input :name
      f.input :location
      f.inputs :profile_photo
      f.inputs :hello_audio
    end
    f.actions
  end
end

We just added the fields. Restart your server and access your users page.
Create a user if you need to.
Notice that on index, show, and edit that the fields for profile_photo and hello_audio are blank. This is to be expected - we haven't uploaded anything yet!

Upload a small image and a small audio clip.

Notice that after we've uploaded it you can only view the path names and not anything inside the file.

To include our 'library', we only need make one small change, in config/application.rb:

require_relative 'boot'
require 'rails/all'
Bundler.require(*Rails.groups)

module Blogly
  class Application < Rails::Application
    config.autoload_paths += %W(#{config.root}/lib) # Add this line!
  end
end

Now, simply change your admin/user.rb file to this final code:

ActiveAdmin.register User do
  permit_params :name, :location, :profile_photo, :hello_audio
  # Show the essential data in the index
  index do
    extend EnhancedUploader # include uploader
    selectable_column
    column :id
    column :name
    column :location
    column :profile_photo do |row_object|
      image_content row_object, :profile_photo
    end
    column :hello_audio do |row_object|
      audio_content row_object, :hello_audio
    end
    column :created_at
    column :updated_at
    actions
  end

  # When you click on the show for any individual item, this data is rendered
  show do
    extend EnhancedUploader # include uploader
    attributes_table do
      row :id
      row :name
      row :location
      row :profile_photo do |item|
        image_content item, :profile_photo
      end
      row :hello_audio do |item|
        audio_content item, :hello_audio
      end
      row :created_at
      row :updated_at
    end
  end
  # When you click on edit, this form is rendered
  form do |f|
    extend EnhancedUploader # include uploader
    f.semantic_errors
    f.inputs do
      f.input :name
      f.input :location
      input_image f, :profile_photo
      input_audio f, :hello_audio
    end
    f.actions
  end
end

Refresh your server, and head to the users page. Now you should see previews of the audio and image that you uploaded on each index, show, and edit page! Success!

Serverless Backends with AWS Cloud: Cloudfront and WAF

Rob Sherling — Mon, 24 Apr 2017 19:06:44 +0000

This is a part in a series on AWS serverless architecture. The original blog post series can be found both here and on my blog J-bytes.

CloudFronts and Firewalls

Why Cloudfront for an API?

I'm gonna be honest: When my boss told me to put the API on CloudFront, I thought he might have been joking. Why would you need to put an API on a distribution network? Setting it up was the hardest part of this project (no, seriously), and working with the quirks of CloudFront was a pain.

Then it became super obvious why we did it once it started working.

Do you remember at the beginning when I said that we were assuming the site you were using this with was a collection of static pages (although it doesn't have to be)? Well, if you put your API on CloudFront and your site in S3, they are all under the same SSL and distribution settings. This means no separate configurations for anything, you can ensure all requests are automatically HTTPS, etc., and the configurations are all in one convenient panel. Also, thanks to WAF, your site and your API are protected in the exact same way. You can never launch the site but forget to make the API available and cause user confusion.

We'll be making (as usual) two API CloudFront distributions - one for each of our stages.

Note: As of the time of this publication, it has been my experience and the experience of several others that cloudfront will randomly drop API requests for no reason. This behavior is intermittent but definitely present (source).

S3 Static Sites

First, however, we need to make an S3 distribution. This is because the first origin that we create in our distribution must have a Path Pattern of Default (*) and catches all routes, and we don't want our API to do that. We want our /api requests to go to our API and the rest to go to our S3. Basically, this is just a placeholder so we'll just rig something up lightning quick.

Go make a new bucket in S3 called "whatever unique name you want here-site." We will now call this our site bucket. Edit the properties of this bucket, and set Static Website Hosting to Enable website hosting, then set the Index Document to index.html. Leave Error Document blank. (source)

Upload a single file into it called index.html with some text of your choosing (I always choose the word delicious for really circuitous reasons based on a Japanese word for a bad situation). Make sure to set the permissions on this file to Everyone - Open/Download. If correctly done, you should be able to open just the file and see everything inside it easily. Let's put that on a distribution really fast, and then get to the good part. Read the following very carefully, because redeploys can take a long time so troubleshooting becomes very costly in terms of time.

Go to the CloudFront panel
Click Create Distribution
Under Web, click Get Started
In Origin Domain Name, type the site bucket name configured as follows: <bucket-name>.s3-website-<aws-region>.amazonaws.com.(source)
Leave Origin Path blank, Origin ID as is, and Origin Custom Headers blank.
For the Viewer Protocol Policy, change it to "Redirect HTTP to HTTPS"
For everything else in Default Cache Behavior Settings, you can leave it as-is. If you need to confirm a lot of rapid site changes in real time on your staging environment, set your Object Caching to Customize and all your TTL to 0. For your Distribution Settings, configure your Price Class to your region. For example, I'm in Japan so we use "Only US, Canada, Europe, and Asia." You probably don't actually have to do this, but it's a great habit to get into. We haven't made a WAF yet (we'll talk about that briefly later), so just leave that blank.
Ignore everything else and scroll down to Enable IPv6. Turn that off unless you know you need it. For example, if this is the backend for an iPhone App, their testing standards require ipv6 connectivity (source).
Under comment, write STAGING.
Create Distribution.
Make sure that your Origin Protocol Policy says "HTTP Only" under the Origins tab. I do not know why, but one of my stages defaulted to HTTPS and it broke my distribution later on. If your screen does not look like this: click on the edit button and simply click Yes, Edit to save your changes. This should hopefully force it back to HTTP only, as it did for me.

Putting the API behind CloudFront

Click on the distribution you we just made, then click the Origins tab and click Create Origin.
In Origin Domain Name, type the base URL for your API. For example, if you go to the API panel and click on Stages, then staging you should see an invoke URL like https://<letters>.execute-api.us-west-1.amazonaws.com/staging. Copy the URL, without the /staging, and paste it in the Origin Domain Name box https://<letters>.execute-api.us-west-1.amazonaws.com
In Origin Path, write the stage name /staging. This is why we didn't write staging above - we automatically append /staging to all api requests that come to this domain.
Leave Origin ID and Origin SSL Protocols as-is
Change Origin Protocol Policy to HTTPS Only. The API Gateway will only accept HTTPS.
HTTPS Port is 443 (as-is)
Origin Custom Headers is as-is
Click Create
Click the Behaviors tab.
Create Behavior.
Path Pattern: /api*. This will match anything that looks like https://<url.top>/api/endpoint. This is also the reason that our API Gateway is set up so all the requests are under a /api resource, because CloudFront will automatically append the path pattern as-is when it send it to the API Gateway. So, as a quick explanation, if we set the CloudFront path pattern to /api* but our gateway was just /email and then tried site.com/api/email, it would never work because CloudFront forwards the /api/email as is to our API Gateway.
Change Origin to the one we just made: Custom-letters.execute-api-stuff
Viewer Protocol Policy is redirect HTTP to HTTPS, because the Gateway demands it.
Allowed HTTP Methods are GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE because we need to post info to the gateway.
Forward Headers are set to Whitelist, then type "Origin" in the box and "Add >>" it to the whitelist. This is required by gateway.
For our Object Caching, set it to customize, and then all TTL must be set to 0. You do not want any caching on the API. This will make requests just plain not work if you try to add GET api methods later.
Forward Cookies -> All
Query String Forwarding and Caching -> Forward All, Cached Based on All
Everything else is no. Create. Now if you go back to your Cloudfront Overview Page, you should see something like this on status: That can, and almost certainly will, take at least 20 minutes to update. While that's working, go ahead and follow all the steps above again, except this time create a production distribution (or don't, I'm not your boss) :). If you do, feel free to point the S3 part at the same bucket because it's just a demo. While we're still waiting (seriously, it takes a while) I'll explain WAF.

WAF: Wireless Application Firewall

You would use a WAF in your actual setup to protect your site. For example, you would configure your WAF for staging so that only your IP Address (and the IP address of your client, if they have to approve site changes before pushing to production) would have access. You can add your production site to that same WAF (for example, before release or during maintenance) and then simply remove it from the WAF when you are ready to launch / finished maintenance.

Go to WAF & Shield in your console.
Click on Go to AWS AWF, then click Configure web ACL. Read the Concepts Overview if you want, then click Next.You probably don't actually have to make a WAF, so unless you have a static IP address or just want to test it, feel free to simply follow along for the learning experience and then delete the result.

The ACL name should be something unique like " Firewall"
Region is global (Cloudfront)
Associated Resource is Production (for now).
Click Next
Click "IP match condition"
Name is Allow Example IP ( for example)
IP version is V4
Address is whatever Google tells you your IP is, followed by a /32(example: 127.0.0.1/32). If you own a range, add it with /24 (see the advice on the page). Feel free to add as many IP addresses as you want to allow. Note that unless you purchase a fixed IP address from your ISP, you cannot count on this number not changing.
Click the Add IP address or range button. If you don't, you'll create an empty condition that will do nothing at all.
Click Create.
Click Next.
On the Create rules page, click Create rule
Name: Allow Example IP (or really, whatever you want). Under conditions, it should read: When a request 'does originate from an ip address in Allow Example IP' Note: If you need to add multiple IPs, do not click "Add condition" and add an IP on this screen. If you do, it will make the condition "All requests coming from IP set 1 and IP set 2 at the same time", which is impossible and invalidates this rule completely. If you created multiple IP conditions in the last page (for example, a set of IP addresses to represent your company and another set to represent your client), you need to create a separate rule for each set.
Click create.
Change the Default Action to "Block all request that don't match any rules." The page should look like this: This tells us that our allowed IPs have been set to allow, and if it isn't on our IP list, block it.
Click Review and create
Confirm that your production Cloudfront is attached at the bottom.
Click Confirm and Create If you want to add your staging distribution to the WAF, it's as easy as clicking Add association on the right-hand side of the screen and selecting your staging distribution. Continue to wait as your Cloudfront Distributions update. Watch Youtube, play a game, etc. When the status is no longer in progress, proceed to testing.

Testing the Cloudfront Distribution and Firewall

Here comes the coolness. Go to your Cloudfront page. You should see both distributions there. Copy and paste the url in the Domain Name section into your browser, and hit enter. Assuming that you configured everything correctly and are coming from the correct IP if you are behind a WAF, you should be seeing the contents of your index.html page. If you try to go to that same domain though a different IP (smartphone on mobile data), you should see a forbidden page. Success!

Try sending some requests in postman to each API (staging and production) with domain-name/api/emails (or twitter) and see what happens. Remember to leave out /staging and /production, as they are included in the origin. You should have the same results as before, and database records should be correctly updated.

That's it for the web site! That was a crazy ride. Next, we'll see what happens when we need to get that data from Dynamodb into our local environment and save emails for clients / send tweets!

Serverless Backends With AWS Cloud: Twitter Sessions

Rob Sherling — Mon, 24 Apr 2017 19:06:44 +0000

This is a part in a series on AWS serverless architecture. The original blog post series can be found both here and on my blog J-bytes.

How to make a session in DynamoDB

In this section, we're just going to save some browsing information in a session (held in DynamoDB) and redirect the user to Twitter. Once we have a callback in place, we'll use that session data to have our user follow us. We can also use that data to message that user later (we actually have the user DM themselves, explained in the next section) with some promotional news.

More specifically, we're going to save their request token in a table in DynamoDB along with some other stuff, then when they confirm our app, use their oAuth token to go get the data attached to the request token and complete the chain.

This is fairly straightforward so let's just jump right into it.

Step 0: Make a Twitter App

Setup your Twitter app here. It's fast, free, and easy. You will need this to do anything cool like message users and create friendships (on Twitter, this technology is not guaranteed to create friendships in any other way), so making it now makes sense. When you set this app up, it is super hyper critical imperative that you go to the permissions tab and select read, write, and access direct messages. You do this because we need to have them DM themselves later on.

Step 1: DynamoDB Tables

You should know the drill from the last time we touched tables, so let's just rig up the rest of our tables right now. We're going to need an additional 4 tables total. We'll need staging and production tables for the session we'll create to hold our user's data while they log in to their Twitter accounts to (hopefully) authorize us, and we'll need staging and production tables to hold their data once they come back from that authorization and hit our callback.

Open the DynamoDB table and make the following tables:

name: staging_session -> primary key: request_token (string)
name: production_session -> primary key: request_token (string)

name: staging_twitter -> primary key: uid(Number)
name: production_twitter -> primary key: uid(Number)

Quick note on the uid in the Twitter tables: The uid actually will be treated as a string in our lambdas and automatically converted on storage. This is because Javascript sometimes has a problem with numbers and uids(source), but DynamoDB does not care at all.

Step 2: env-config.json

Open up - or download from your S3 - your env-config.json file (you should have two, one for each environment we are creating). We'll be filling in the rest of the values, leaving only the twitter_api_callback blank for now (we'll do that in the next section). You can get your consumer_key and consumer_secret values from your Twitter app that you set up in step zero. You can find it in Keys and Access Tokens tab on your app settings.

The twitter_table_name should be <stage>_twitter, and the twitter_session_table_name is <stage>_session.

For the follow uid, you can get it from the same Keys and Access Tokens tab labelled as Owner Id. If it is not your own account use this link and put in the account that you want to follow.

Save your JSON, and re-upload to the appropriate S3 buckets.

Step 3: Constructing the Twitter Lambda

Make a session_lambda folder on your local disk, and cd into it.

We're going to download libraries like we did before. This time, we'll need the node-twitter-api library. Observe the spacing on this command, it's a little tricky (careful when you copy):

npm install –prefix=./ node-twitter-api

Add the loader.js that we made before using either a hard system link or copying it into the folder, depending on your expertise. If you're copying the loader.js from the github repo for the project, remember to change the S3 bucket on line 61 to our actual bucket.

Add the index.js file.
Change the line on line 6 to your appropriate db region.

Read the index file for the notes, but it should be fairly straightforward. We're just going to take a POST request that will tell us if the user chose to follow us or not, hit the Twitter API to get some request tokens, save that data all together, and then redirect to the Twitter site so they can complete the authorization. If you don't need users to follow an account or track any other data in the session, feel free to make it a get request later on.

Zip the folder like we did when we made the email lambda (zip the contents of the folder, not the folder itself) and head on over to the lambda console.

Create a lambda function. Select Blank Function for the blueprint.
Click Next on the Configure Triggers section that pops up to move to the next screen.
For the name of the function, use generate_twitter_session.
Change the Code entry type drop-down box to "upload a zip file."
Upload the zip we just made.
Under Lambda function handler and role, use master_lambda in the Existing role dropdown box.
Set the Timeout to 15s
Click Next, then Create Function.
From the Actions drop-down, configure the test event.

{
 "body-json": "follow=true"
}

Click Save.
Click Actions and create two aliases. One 'staging', the other 'production'. Both should point to Version: $LATEST (you can make a version 1 if you'd like).
Click the button to the left of test (could start with either Alias or say Qualifiers) and click staging under the aliases drop-down tabs. Then click test.
If you get an error message in your logs on a line with JSON.parse that says SyntaxError: Unexpected token u or something similar, check your test event again.

You should get an error message that says:

  "errorMessage": "Twitter.MovedPermanently : Redirecting.",
  "errorType": "https://twitter.com/oauth/authenticate?oauth_token=<SOME TOKEN STUFF HERE>"
If you do, you are in good shape and ready to move on to hooking up our API endpoint.

Step 4: Making our Gateway

We're just going to hook up an API endpoint to our Twitter. This is basically a carbon-copy of the email API we made, so if you need a refresher please go read that again now.

Open the API Gateway console and click on our services-api that we made. Click on /api, and then click the Actions drop-down and select Create Resource. Name that resource twitter-session, and click Create Resource.

This resource is going to mimic our email resource nearly identically. Remember to have your AWS CLI ready to approve our stage variable controlled lambdas.

Click on the twitter-session resource, and click create method. Change that method to post, and click the check mark.

Change the lambda region to your region, then type generate_twitter_session:${stageVariables.alias}. Save, and when you see that giant wall of text, put it into the AWS CLI and replace ${stageVariables.alias}with staging. Enter, then replace it again with production. Enter again. Confirm non-error text.

Back in your lambda console, click okay. Then click on Method Response. Delete the 200, add a 301, and then edit it to include the 'location' header.

Go to the integration response. Delete the default response with a status of 200. Click add integration response, change the status to 301, and make the error regex, ^Twitter.MovedPermanently.*

Edit our newly created integration response and set the value of the location header mapping to integration.response.body.errorType. Click the checkmark.

Lastly, go back to our Integration Request from the twitter-session post option, and add a body mapping template. The type is application/x-www-form-urlencoded. Set the general template to Method Request passthrough, and the Request body passthrough to When there are no templates defined. Save it all.

Deploy to staging and production. Export. Save.

Step 5: Postman

Click on staging under stages, then click on /api/twitter-session. Copy the invoke URL and paste it into Postman. Set the header to Content-Type: application/x-www-form-urlencoded. Set the body to x-www-form-urlencoded, and the data key-value to follow : true. Ensure that the method type is POST, and hit send. Postman should say loading for a short bit (if, for example, you are doing this from near Tokyo when your server is in California, it may be a long bit), and then come at you with something like this (you will have to click "pretty" to make it pretty):

It works!

There is no need to go any farther than this for now. If you check your staging_session database table, you should have some data in it. That marks the completion of this section.

In the next section, we'll send confirmation tweets, actually follow an account, etc.

DEV Community: Rob Sherling

React Native PSA - select and highlight text with custom context menus

What is this?

Repo

Relevant info

iOS Certificates and Profiles in 5 minutes

Certs and Profiles: concept

Certificates

Profiles

Actual practical example

My AWS service was over-provisioned by ~1000 times: A 5-minute guide on how to check app resource use on fargate

How to actually profile your production app

Memory Usage (in MiB)

CPU Usage

Working in Japan 2: Freelancing, Job Hopping, More Salary, Visa Types, Learning Japanese

Table of Contents

Types of freelance

Freelance company

Direct contract

Companies that aren't freelance but look like it

No need to network

Benefits over being a regular employee

Salary differences

Meritocracy

Remote work is easier to get

Fewer meetings

Easy access to cooler problems

No code tests

So much time off

Flexibility in work hours

True freedom

One interview

Job hopping isn't a big problem

Downsides

No stability

No friends

Language

Skill

Benefits

Compensation

How much to charge?

Health insurance

Taxes

Bonus #1 - ふるさと納税

Bonus #2 - American? Read this

Visa - Highly skilled professional visa

Normal work Visa

Learning Japanese

Full explanation

Very Short Version

An app to help you learn

Conclusion

I'm a programmer changing jobs in Japan. This is a list of all the stuff I've learned on my job hunts.

Questions

Question set #1: What do you, personally, like the most about the company? What do you want to improve the most? (do not use the word dislike)

Question #2: How many people have quit in the last year / what percentage of people quit this year?

Question #3: What are your hobbies?

Question #4: Do you have a process for meetings?

Question #5: When is the code interview?

Question #6: What did you do before this, and why did you change?

Advice

Advice #1: Do not write most things down.

Advice #2: Don't smash yes on each casual interview offer.

Advice #3: Never talk salary with anyone who isn't HR/manager at final stage

Advice #4: Salary discussion in the first interview should raise suspicion.

Advice #5: The questions they don't answer are the most important.

Advice #6: Imagine being a Toxic Person

Advice #7: Pay attention to your gut.

Working in Japan: Myths, Realities, Salary, Culture (By A Software Engineer)

Programming in Japan

Table of Contents

Intro

Qualifications

Starting My Career Here

Working Conditions

Hours

Paid Leave

Contracts And Rules

Working in Games

Company Rigidity and Change