DEV Community: Zia Ul Rehman

Story of an SQL query ~10x speed-up with a simple tweak

Zia Ul Rehman — Thu, 30 Nov 2023 19:15:08 +0000

Let’s re-visit basics of debugging performance issues and importance of monitoring tools(focusing on web apps), with a story of simple tweak improving a bottleneck query performance up-to 10x

TL;DR

A walk-through of simple debugging process for performance issues, and how simple tweaks can make your code run so much faster with minimal efforts to spare on performance. (80/20 rule)

Where to start fixing bad performance

So you have bad performing app or pages, now what?

There are two obvious paths , as you being the creator or maintainer of the app, you already have basic idea of where the bottle necks are, which pages load slow etc, so you can just go and try fixing things, This may work for smaller apps, but of-course, this is not a systematic way.

A good approach for most production apps, is to have things setup in a systematic manner , have performance and server monitoring setup done right(don’t over kill), collect data, analyze data and quickly fix the issues with the richness of insights you get from the right tooling.

I suggest almost always going with the systematic manner, no matter the size of the app. If you have long term plans with you app that is.

Performance monitoring tooling

I am assuming you have an app already in production of test phase, which is facing memory bloat issues or slow API response times, where do you start with? Of-course, you need tooling that can help you determine not only the slowness causing areas in your system, but also the breakdown of insights like how much time DB is taking, how much time each function is taking and so on.

We have many tools for this, my personal favourites, new relic APM, and Skylight(it focuses on Ruby on Rails apps, so has more niche offerings)(not affiliated to any in any way). I use both regularly. Of-course you can use any APM , and even 1 single tool is enough, what you need at the end is:

Ability to see slow end points or processes based on a few parameters:
1. _ Slowest average response time _ (transactions which perform the worst)
2. _ Most time consuming _ (transactions which collectively takes most of the server time)
3. _ Throughput _ (most popular transactions)
Ability to provide insights into each of the server transaction(we define a transaction as being a unit of work, be it an API call or a background job)
1. _ Time being taken _ by each section of the code
2. Total time spent on _ DB vs in app _ processing
3. Break down of all the SQL queries, and their run time in the transaction
4. Aggregation of these insights, like overall slowest queries in the system etc
In addition, some tools provide even granular insights by trying to identify N+1 queries, repeated queries patterns and showing exactly how much time is spent on each query (i find skylight very good at this)
Also, another really helpful thing, especially when debugging memory issues, is tooling which can tell us how much memory allocations are done in each section of the code , being run in a single transaction, and of-course, total average allocations per transaction. (again, i find skylight better in this regard)

If we have our tooling setup right, and we are getting as much insights as we need, tool name at the end doesn’t matter as such. It’s very much critical to have your setup right! Everything else, is on your taste and maybe somewhat dependent on your framework.

Just one thing is to ensure working in the right direction, i.e focusing on high impact or frequent transactions instead of just trying to fix the slowest one , that might not even be called in days.

After Identifying Slowest Transactions

Now after all the basic setup, you have your slowest transactions at hand, you pick single transaction you want to focus on, based the impact of that transaction(like an endpoint directly impacting core function of the app.

With right tooling, you should have the all the data you need (even code section level insights) to help you find bottle-necks in slow transactions.

We don’t intend going to a lot of details of each type of fixes you can make with these insights, any engineer can use this data and build upon this information their hypothesis, verify them, and than fix the issues they have identified.

Only one thing i would like to highlight is the importance of benchmarking.

Benchmarking your fixes

Benchmarking in this context is having a base performance stats logged, and than monitoring the performance of same transactions after you fix them, to see how they actually perform and stand against the previously logged transaction performance

Simple SQL query tweak improved it ~10x

(We will mask some details as to ensure code privacy)

Also, we will see why i mentioned 80/20 rules on top, and how it helps us move fast and focus on what matters most and create most impact with minimal efforts.

We had an endpoint in our app, which was fetching Posts i am associated with in some manners, say we have post table and associated tables like post_statuses, post_reports, post_events, post_collaborators and so on. This endpoints fetches all posts where :

Post is in specific statuses (published, draft etc)
Post has any pending reports assigned to current user for review (abuse reports)
Current user is collaborator in the post

Plus, all of these posts should be ordered in a very specific order, with some data on top for each report, i.e:

Order by latest activity (activity is defined across multiple tables’ operations)
Include unread comments count (to show kind of like whatsapp shows unread message count)

Now this defines the baseline of the structure and requirements, of-course there are more details to the matter, and the long single SQL query, taking like 25-30 seconds per request.

Original Query

Looked something like:

SELECT posts.*, COUNT(prs.id) AS pending_reports, posts.*, 
GREATEST(MAX(post_comments.created_at), MAX(post_events.created_at), GREATEST(posts.created_at, posts.updated_at)) AS latest_activity,
count(DISTINCT(user_comments.id)) FILTER (where user_comments.id IS NOT NULL) AS unread_count,
GREATEST(MAX(comments.created_at), posts.created_at) AS last_activity 

FROM "posts"
INNER JOIN "post_statuses" ON "post_statuses"."post_id" = "posts"."id" 
LEFT JOIN post_reports as pr ON pr.reportable_id = posts.id AND pr.reportable_type='Post' 
LEFT JOIN post_report_statuses as prs ON prs.post_report_id = pr.id AND pr.active = true AND pr.user_id = 1 
LEFT JOIN comments AS post_comments ON post_comments.commentable_id = posts.id AND post_comments.commentable_type = 'post' 
LEFT JOIN post_events ON post_events.post_id = posts.id 
LEFT JOIN comments ON comments.commentable_id = posts.id AND comments.commentable_type = 'post' 
LEFT JOIN user_comments ON user_comments.comment_id = comments.id AND user_comments.user_id = 1 AND user_comments.read_at IS NULL 
LEFT JOIN post_collaborators as pco ON pco.post_id = posts.id AND pco.user_id = 1 AND pco.post_signed_off = 'f' 
WHERE "posts"."id" IN (
     SELECT DISTINCT "posts"."id" 
     FROM "posts" 
     LEFT JOIN post_collaborators AS current_post_collaborators ON current_post_collaborators.post_id = posts.id AND current_post_collaborators.deleted_at IS NULL 
     LEFT JOIN post_reports AS user_reports ON posts.id = user_reports.reportable_id AND user_reports.reportable_type = 'post' 
     LEFT JOIN post_report_statuses ON user_reports.id = post_report_statuses.post_report_id AND post_report_statuses.status = ANY(ARRAY [0, 1, 2]) 
     WHERE (posts.moderator_id = 1 OR current_post_collaborators.user_id = 1 OR user_reports.user_id = 1 )
) AND (
"posts"."creator_id" IN (SELECT "users"."id" FROM "users" WHERE ((users.first_name || ' ' || COALESCE(users.last_name, '')) ILIKE '%%')) 
OR "posts"."moderator_id" IN (SELECT "users"."id" FROM "users" WHERE ((users.first_name || ' ' || COALESCE(users.last_name, '')) ILIKE '%%'))) 
GROUP BY "posts"."id" 
ORDER BY latest_activity DESC, last_activity DESC 
LIMIT 50 OFFSET 0

At first glance, there are some obvious problems in this SQL , remember this SQL is produced by an ORM, not directly written by a developer, and ORMs while make things easy and fast to build, sometimes can introduce un-intended complexity.

Original Query Plan

Here is how its Query Plan with Explain Analyze looked like this(don’t try to read it in full)

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit (cost=237736.02..237736.15 rows=50 width=694) (actual time=27772.897..27772.922 rows=50 loops=1)
   -> Sort (cost=237736.02..237785.96 rows=19974 width=694) (actual time=27772.895..27772.916 rows=50 loops=1)
         Sort Key: (GREATEST(max(post_comments.created_at), max(post_events.created_at), GREATEST(posts.created_at, posts.updated_at))) DESC, (GREATEST(max(comments.created_at), posts.created_at)) DESC
         Sort Method: top-N heapsort Memory: 121kB
         -> GroupAggregate (cost=33452.05..237072.50 rows=19974 width=694) (actual time=373.867..27771.558 rows=685 loops=1)
               Group Key: posts.id
               -> Merge Left Join (cost=33452.05..226439.26 rows=685580 width=371) (actual time=369.564..27562.155 rows=1371572 loops=1)
                     Merge Cond: (posts.id = pco.post_id)
                     -> Merge Join (cost=32483.89..223659.06 rows=685580 width=371) (actual time=368.498..27302.152 rows=1371572 loops=1)
                           Merge Cond: (post_statuses.post_id = posts_1.id)
                           -> Merge Left Join (cost=27619.84..193204.04 rows=7477059 width=379) (actual time=104.873..22591.305 rows=71784103 loops=1)
                                 Merge Cond: (posts.id = post_events.post_id)
                                 -> Merge Left Join (cost=27619.55..70066.69 rows=1138329 width=371) (actual time=104.821..1694.553 rows=4350298 loops=1)
                                       Merge Cond: (posts.id = post_comments.commentable_id)
                                       -> Merge Left Join (cost=27619.25..37371.24 rows=262806 width=363) (actual time=104.790..336.002 rows=379658 loops=1)
                                             Merge Cond: (posts.id = comments.commentable_id)
                                             -> Merge Join (cost=2181.16..7665.76 rows=60674 width=347) (actual time=4.289..89.766 rows=79948 loops=1)
                                                   Merge Cond: (posts.id = post_statuses.post_id)
                                                   -> Merge Left Join (cost=2180.74..4579.45 rows=19974 width=339) (actual time=4.271..44.332 rows=26882 loops=1)
                                                         Merge Cond: (posts.id = pr.reportable_id)
                                                         -> Index Scan using posts_pkey on posts (cost=2096.29..4435.40 rows=19974 width=331) (actual time=3.439..35.952 rows=26630 loops=1)
                                                               Filter: ((hashed SubPlan 1) OR (hashed SubPlan 2))
                                                               SubPlan 1
                                                                 -> Seq Scan on users (cost=0.00..1043.38 rows=1850 width=8) (actual time=0.020..3.018 rows=1850 loops=1)
                                                                       Filter: ((((first_name)::text || ' '::text) || (COALESCE(last_name, ''::character varying))::text) ~~* '%%'::text)
                                                               SubPlan 2
                                                                 -> Seq Scan on users users_1 (cost=0.00..1043.38 rows=1850 width=8) (never executed)
                                                                       Filter: ((((first_name)::text || ' '::text) || (COALESCE(last_name, ''::character varying))::text) ~~* '%%'::text)
                                                         -> Sort (cost=84.45..86.39 rows=773 width=12) (actual time=0.826..1.038 rows=777 loops=1)
                                                               Sort Key: pr.reportable_id
                                                               Sort Method: quicksort Memory: 61kB
                                                               -> Hash Right Join (cost=29.62..47.37 rows=773 width=12) (actual time=0.463..0.673 rows=777 loops=1)
                                                                     Hash Cond: (prs.post_report_id = pr.id)
                                                                     Join Filter: (pr.active AND (pr.user_id = 1))
                                                                     Rows Removed by Join Filter: 775
                                                                     -> Seq Scan on post_report_statuses tas (cost=0.00..15.72 rows=772 width=16) (actual time=0.005..0.078 rows=779 loops=1)
                                                                     -> Hash (cost=19.96..19.96 rows=773 width=21) (actual time=0.321..0.322 rows=777 loops=1)
                                                                           Buckets: 1024 Batches: 1 Memory Usage: 57kB
                                                                           -> Seq Scan on post_reports ta (cost=0.00..19.96 rows=773 width=21) (actual time=0.014..0.204 rows=777 loops=1)
                                                                                 Filter: ((reportable_type)::text = 'post'::text)
                                                                                 Rows Removed by Filter: 24
                                                   -> Index Only Scan using index_post_statuses_on_post_id on post_statuses (cost=0.29..2265.14 rows=80899 width=8) (actual time=0.015..27.638 rows=79948 loops=1)
                                                         Heap Fetches: 28565
                                             -> Materialize (cost=25438.10..26014.87 rows=115355 width=24) (actual time=100.494..154.413 rows=370263 loops=1)
                                                   -> Sort (cost=25438.10..25726.48 rows=115355 width=24) (actual time=100.490..120.023 rows=115303 loops=1)
                                                         Sort Key: comments.commentable_id
                                                         Sort Method: external merge Disk: 2952kB
                                                         -> Hash Left Join (cost=4747.85..13373.21 rows=115355 width=24) (actual time=4.067..61.603 rows=115304 loops=1)
                                                               Hash Cond: (comments.id = user_comments.comment_id)
                                                               -> Seq Scan on comments (cost=0.00..8322.54 rows=115355 width=24) (actual time=0.006..39.808 rows=115304 loops=1)
                                                                     Filter: ((commentable_type)::text = 'post'::text)
                                                                     Rows Removed by Filter: 27899
                                                               -> Hash (cost=4734.53..4734.53 rows=1066 width=16) (actual time=4.054..4.055 rows=807 loops=1)
                                                                     Buckets: 2048 Batches: 1 Memory Usage: 54kB
                                                                     -> Bitmap Heap Scan on user_comments (cost=39.41..4734.53 rows=1066 width=16) (actual time=0.567..3.899 rows=807 loops=1)
                                                                           Recheck Cond: (user_id = 1)
                                                                           Filter: (read_at IS NULL)
                                                                           Rows Removed by Filter: 2899
                                                                           Heap Blocks: exact=1935
                                                                           -> Bitmap Index Scan on index_user_comments_on_user_id (cost=0.00..39.15 rows=3030 width=0) (actual time=0.270..0.270 rows=3889 loops=1)
                                                                                 Index Cond: (user_id = 1)
                                       -> Materialize (cost=0.29..17628.74 rows=115355 width=16) (actual time=0.027..391.368 rows=4340903 loops=1)
                                             -> Index Scan using index_comments_on_commentable_id on comments post_comments (cost=0.29..17340.35 rows=115355 width=16) (actual time=0.024..117.440 rows=115303 loops=1)
                                                   Filter: ((commentable_type)::text = 'post'::text)
                                                   Rows Removed by Filter: 9612
                                 -> Materialize (cost=0.29..8572.97 rows=174931 width=16) (actual time=0.049..4195.281 rows=71784103 loops=1)
                                       -> Index Scan using index_post_events_on_post_id on post_events (cost=0.29..8135.64 rows=174931 width=16) (actual time=0.047..97.077 rows=173202 loops=1)
                           -> Unique (cost=4864.04..4876.19 rows=2430 width=8) (actual time=53.422..53.979 rows=685 loops=1)
                                 -> Sort (cost=4864.04..4870.12 rows=2430 width=8) (actual time=53.419..53.623 rows=904 loops=1)
                                       Sort Key: posts_1.id
                                       Sort Method: quicksort Memory: 25kB
                                       -> Hash Left Join (cost=1873.16..4727.40 rows=2430 width=8) (actual time=11.725..53.306 rows=904 loops=1)
                                             Hash Cond: (posts_1.id = user_reports.reportable_id)
                                             Filter: ((posts_1.moderator_id = 1) OR (current_post_collaborators.user_id = 1) OR (user_reports.user_id = 1))
                                             Rows Removed by Filter: 70531
                                             -> Hash Right Join (cost=1813.22..3964.41 rows=66906 width=24) (actual time=10.900..43.152 rows=70403 loops=1)
                                                   Hash Cond: (current_post_collaborators.post_id = posts_1.id)
                                                   -> Seq Scan on post_collaborators current_post_collaborators (cost=0.00..1975.53 rows=66906 width=16) (actual time=0.016..12.351 rows=66718 loops=1)
                                                         Filter: (deleted_at IS NULL)
                                                         Rows Removed by Filter: 23609
                                                   -> Hash (cost=1480.32..1480.32 rows=26632 width=16) (actual time=10.868..10.869 rows=26632 loops=1)
                                                         Buckets: 32768 Batches: 1 Memory Usage: 1505kB
                                                         -> Seq Scan on posts posts_1 (cost=0.00..1480.32 rows=26632 width=16) (actual time=0.009..6.641 rows=26632 loops=1)
                                             -> Hash (cost=50.28..50.28 rows=773 width=12) (actual time=0.733..0.735 rows=779 loops=1)
                                                   Buckets: 1024 Batches: 1 Memory Usage: 45kB
                                                   -> Hash Right Join (cost=29.62..50.28 rows=773 width=12) (actual time=0.306..0.625 rows=779 loops=1)
                                                         Hash Cond: (post_report_statuses.post_report_id = user_reports.id)
                                                         -> Seq Scan on post_report_statuses (cost=0.00..18.62 rows=772 width=8) (actual time=0.009..0.136 rows=779 loops=1)
                                                               Filter: (status = ANY ('{0,1,2}'::integer[]))
                                                         -> Hash (cost=19.96..19.96 rows=773 width=20) (actual time=0.289..0.290 rows=777 loops=1)
                                                               Buckets: 1024 Batches: 1 Memory Usage: 51kB
                                                               -> Seq Scan on post_reports user_reports (cost=0.00..19.96 rows=773 width=20) (actual time=0.009..0.181 rows=777 loops=1)
                                                                     Filter: ((reportable_type)::text = 'post'::text)
                                                                     Rows Removed by Filter: 24
                     -> Sort (cost=968.16..968.80 rows=254 width=8) (actual time=1.063..15.024 rows=245069 loops=1)
                           Sort Key: pco.post_id
                           Sort Method: quicksort Memory: 25kB
                           -> Bitmap Heap Scan on post_collaborators tus (cost=358.00..958.02 rows=254 width=8) (actual time=0.871..1.038 rows=138 loops=1)
                                 Recheck Cond: (user_id = 1)
                                 Filter: (NOT post_signed_off)
                                 Heap Blocks: exact=113
                                 -> BitmapAnd (cost=358.00..358.00 rows=254 width=0) (actual time=0.854..0.855 rows=0 loops=1)
                                       -> Bitmap Index Scan on index_post_collaborators_on_user_id (cost=0.00..10.06 rows=769 width=0) (actual time=0.055..0.055 rows=741 loops=1)
                                             Index Cond: (user_id = 1)
                                       -> Bitmap Index Scan on index_post_collaborators_on_post_signed_off (cost=0.00..347.56 rows=29769 width=0) (actual time=0.779..0.779 rows=30347 loops=1)
                                             Index Cond: (post_signed_off = false)
 Planning Time: 5.432 ms
 Execution Time: 27773.891 ms
(108 rows)

It is obviously too long to read, but there are some key areas like :

Actual cost and planned cost to look closely into, especially where it is highest.
Lookout for the sequential scans, and fix them with appropriate indexes.

Start with small and obvious things

But remember we are here following 80/20 rule, we went systematically to find bottlenecks, but we will not go in and fix everything we see not making sense, we will start small, and see impact while we improve. First lets remove the useless blocks at the end which look something like:

(
"posts"."creator_id" IN (SELECT "users"."id" FROM "users" WHERE ((users.first_name || ' ' || COALESCE(users.last_name, '')) ILIKE '%%')) 
OR "posts"."moderator_id" IN (SELECT "users"."id" FROM "users" WHERE ((users.first_name || ' ' || COALESCE(users.last_name, '')) ILIKE '%%')))

These likes are literally doing nothing, they will validate to true for all ALL users, so we don’t really need them, they are intended to be there when there is a free form search on username. But we had a missing check which was causing this SQL to be written even when not needed.

Removing this made a huge impact, like a 500ms impact on a 30s query, lol. But that was a start. Don’t worry.

Remove duplicate join

At this stage, we noticed a duplicate join:

LEFT JOIN comments AS post_comments ON post_comments.commentable_id = posts.id AND post_comments.commentable_type = 'post' 
LEFT JOIN comments ON comments.commentable_id = posts.id AND comments.commentable_type = 'post'

These are essentially joining on same conditions , but one is aliased and one is not. Was not something looked like could make a huge difference, but hey, continue with small fixes.

So i removed one join and re-tried the final query… and we got:

 Planning Time: 9.905 ms
 Execution Time: 2151.695 ms
(93 rows)

This is 2.1 seconds, vs the original 25-30 seconds variant. Voila, 10-15x improvements with 1 simple tweak. This is where we decided that it is good enough to push in first iteration of the enhancements, we opened up our rails models, and fixed a couple of scopes, dried out some SQL in different scopes, and we had this all sorted.

(As we pushed this fix, we continued to monitor the performance benchmarks for respective transactions, and we saw up-to 10x improvements in overall stats of slowest response time, and average response time etc.)

Further optimisations, learning where to stop

At this stage, we have achieved our 80% from the 80/20 rule, we have acceptable performance for this endpoint, i know 2 seconds is still too slow , but this was for worst case, average case is gone under 400ms, which is still not great, but acceptable in our use case(We have other slow endpoints to take care of).

But, what i would like to touch base on, is why 80/20 rule says that 80% of the desired outcomes can be achieved with 20% efforts(in the right direction), and rest 20% of the desired outcomes, take 80% of your time, with showing you how much efforts more optimizations would require.

If you look at the query again, there is a complex select:

GREATEST(MAX(post_comments.created_at), MAX(post_events.created_at), GREATEST(posts.created_at, posts.updated_at)) AS latest_activity

And what’s more is, we are even sorting on this! Which means, for all eligible rows, our DB has to calculate this giant value and than sort on the result of select. Which is obviously going to be very slow.

One solution which comes to mind is, that we can store a calculated latest_activity_timestamp value for posts , and maintain that value in a reasonable manner(via model callbacks or DB triggers), and have it indexed so we can run our queries in the most optimal manner.

BUT, as you can tell, introducing a new calculated value column in DB, and making sure all triggers are set right against this, in going to be a significant effort, compared to what we did before by just removing 2-3 lines of SQL and achieving 10x improving.

I know doing this complex activity, may give us more 10x improvement(bring 2s query time to 200ms), but that is not worth it right now for us, we will do this when it bites us more than other high impact things. We have got 80% of the desired results and we saved plenty of time to spend on other problems.

So lesson here is to learn where to start, where to stop and finding the sweat spot for your use case.

Working around un-maintained redux-token-auth for redux and react 17 upgrade

Zia Ul Rehman — Sun, 26 Dec 2021 12:56:45 +0000

A small case study from our recent efforts to upgrade our legacy react app, which included upgrading and replacing a number of libraries. This one focuses on one of them, that is redux-token-auth.

Tl;DR:

redux-token-auth had been a go to choice for many rails devs who started working with standalone frontends in react. But it has been majorly un-maintained, while on the journey of upgrading our front-end stack, this was becoming a blocker as it was locking to older version of redux and react etc, also had security issues. We ended up with a custom clone where we managed to fix as much as we needed.

Why are we sharing this?

This is a small case study of how we upgraded this package, and what issues we faced, what route we took and where we landed, we share with you what issues faced AFTER the upgrade. And how we resolved them. The purpose of sharing this small story is that before we attempted this upgrade, we tried searching on the internet for this, and didn’t found much, so we had to do our own R&D and it took us ~2-3 days of effort to finalize this upgrade. So if anyone is upgrading their stack(which we should do often) this should save them some time.

Full Story

Intro

redux-token-auth is a great library. What it mainly does is it provides a plug and play auth implementation functionality for ruby on rails based APIs which implement popular devise_token_auth for auth handling.

Problem?

Problem is, redux-token-auth is un-maintained for a long time , says so on readme, has number of open issues, with little to no activity. I don’t work in typescript, otherwise I would LOVE to take it over. (PS: A great opportunity if you want to actively work on an open source project, which is already used a lot, this is looking for maintainers.)

It works fine, for the most part, only when we tried upgrading redux and react, there were strange issues, which when we investigated when upgrading, pointed to multiple major versions of redux and react being in our package-lock, and being used by different components. Which is obviously a problem. There were already open issues for this, with no solution, so we had to dig.

Also there was old version of axios with vulnerabilities , which we had to get rid of.

Fix:

We tried with exploring the existing clones , there were number of clones, as its been years for this package being un-maintained. We looked into many of the clones, mainly looking into change-logs and commit changes from core. Many clones were also fixing 1 or two issues. It seemed like we will have to learn typescript and than do the fixes, which i was not willing as much to do.

After hours of search and try, we found a relatively healthy clone with a LOT of changes from original one, it felt a bit dangerous but we went through the changes_(I don’t do typescript, but i can understand its code, like someone who doesn’t speak a language but can understand it somewhat when spoken to him )_. (it had all changes in develop and not in master/main, which did caused some confusion and overlooking in start.)

This clone had already done the major parts of our work react and redux dependencies were pinned to latest , we just had to upgrade axios, luckily it didn’t cause any breakage in functionality. Then there were some patches we were using, you can see the commit history on github for our final clone as well, but i will put in a summary here as well, to save you some time.

Changes we got from our base clone:

Minimum required Node version from 8 to 10
React from 15 to 16
Redux from 3 to 4
All other dependencies also upgraded including jest/typescipt etc (detail here)
Also a lot of code refactor, majorly because of upgrades of packages

Further changes we made:

Axios upgrade (to fix security issue errors)
General package upgrade (in a bid to make it last a bit longer)
Fixes to remove a patch we were using prior to upgrades, as we use rotating tokens

All good?

Not quiet, we did this and used "redux-token-auth": "git+https://github.com/ziaulrehman40/redux-token-auth.git#9812f831a84968dba593b22b3f48b50b322e2351" in our package.json , but we bumped into this issue, for which we had to take this route.

All good now? Neh, Yet another issue (and a small hack to fix it)

redux-token-auth sets some data in redux under reduxTokenAuth.currentUser key to help us identify if auth attempt is in progress, or has been completed and failed/succeeded. But default data points ( isLoading and isSignedIn ) were not working as expected for some edge cases, after a lot of debugging(and nearly giving up on the whole custom clone thing), i found an un-documented data point, which helped us with everything, it is hasVerificationBeenAttempted, using this and incorporating it in our checks fixed the issue. This is the only part which seemed a bit hacky, but it was to be used in only 1-2 places, so didn’t seem like a big issue.

Next?

Next we moved to upgrading journey for other pieces of our app. And at the time of writing this, we have a final build ready of upgrades ready to go out in a couple of days. We will try to share more of our journey with you guys soon.

If you have any feedback, comments or questions, don’t hesitate at all. Until next time, take care!

Devise gem force confirmation on email change and remove old email

Zia Ul Rehman — Thu, 09 Jan 2020 10:26:18 +0000

Default behavior of devise when confirmable in model and reconfirmable in devise.rb are enabled is that it does send a reconfirmation email email if email is changed, but it does not remove/replace previous email and does allow users to still login through old email while new email’s confirmation is pending.

I am on rails 6.0.2, Devise 4.7.1 for the record.

Problem?

This is a bit odd behavior for some scenarios, like when users are being controlled by admins(lets say in tenants), sometimes we want to enforce email change and reconfirmation when admin changes the email of any user. I could not find any official solution for this issue, and i had to go with monkey patching.

Solution:

For this, i had to overwrite a method in my user model:

  # overwirte from /devise/models/confirmable.rb to force email re-confirmation for email changes
  def postpone_email_change_until_confirmation_and_regenerate_confirmation_token
    @reconfirmation_required = true
    self.unconfirmed_email = email
    # self.email = self.email_was
    self.confirmed_at = nil
    self.confirmation_token = nil
    generate_confirmation_token
  end

And voila, this replaces old email completely and forces a confirmation of that new email before users can continue using the system.

It took me a couple of hours to come to this solution after finding no official solution for this seemingly simple usecase, so logging for my record as well as for community. Happy coding!

Shopify-polaris form building with Formik, React-Final-Form and Yup

Zia Ul Rehman — Sun, 24 Nov 2019 13:56:40 +0000

Disclaimer

Not a code guide of either of the mentioned tools, but rather my experience using these, the problems I faced and the solution we adopted.

TL;DR

We opted Formik to build forms in our app, everything was going fine until we had a long dynamically generated form. With as few as ~100 fields, it started to lag a lot. Than I stumbled upon lot of issues Formik has (both open and closed) about speed issues and double renders, so decided to try something else.

React-final-form was next in the list, It turned out to be great choice with a lot of fine grained control options, speed was a lot better, we also enabled validateOnBlur, which made it lighting fast. We had to build a few wrappers for shipify-polaris components in use. You can find the code/s below.

Long version

We started building a shopify embedded app with Rails 6 and ReactJS. I was using react_rails gem for the purpose of binding the two.

For those of you who don’t know, shopify provides a library of UI components among some guidelines to build shopify plugins/apps keeping the experience as close to original shopify experience as possible.

As shopify plugins are rendered in an iframe, that means anyone can use whatever he wants for the UI/backend and pretty much everything. This can result in completely different UI components and their look and feel across different plugins. This is where shopify-polaris comes in to unify the user experience across different plugins.

Shopify provides official react and html versions of this, but their guidelines can be followed independently of the UI framework.

ENOUGH OF THE SHOPIFY, WHERE ARE FORMS!

OHK! I hear you. Lets get right into forms.

Building forms with formik

Formik is a nice small size library, it is supposed to support all common cases and does not present itself as one stop shop for all kind of form needs. And they are right. We build small forms with relatively simple Yup validation schema and it was working great. There was no lag or anything like that when editing the form.

Than we had to create a dynamic form which could have N sections, and each section will have minimum of ~35 fields.

Building schema for this complex and conditional form was also interesting but that’s not topic here.

Our normal form had around 100+ fields. When we did our development and testing on sample form of 1 section, everything was working fine. We used a small helping library @satel/formik-polaris to bind shopify components and formik without any extra work.

After everything was built, when we tested this against real world loads of forms with 3-4 sections. It showed huge lag when editing. We would see text changing after a complete second of stopping the key-presses. This was obviously un-acceptable to deliver the feature. Here is when our debugging journey began.

Trying to fix lags while sticking with formik

As I mentioned earlier, we had a complex schema with conditional logics, array of objects and so on. We knew this is one place where the bottleneck is, formik is validating whole form on a single key-press, but it did not justify this much of the lag.

One other issue was multi rendering , it was re-rendering whole form minimum of 2 times on single key press. Which of-course means a lot of CPU load.

Memoization

Initially we suspected it is the re-renders which is causing the main issue, so we did split our components in smaller chunks with memoization in mind. And used React.memoize to make them pure and stop their re-renders. but despite moving a huge chunk of form to memoized versions, there was little to no effect on lag.

Try to reduce re-renders to 1

There were multiple issues which we found during our debugging on formik about multiple re-renders, with very few ever resolved, and that either didn’t help us anyway. So we were stuck with multiple re-renders.

At this point we were so frustrated with this experience and saw number of open issues about speed on formik’s large forms that we were fully convinced that formik speed issue is a real thing , and we need to move forward and try something else. This is when I saw a suggestion on an issue comment in formik to use react-final-form and we thought why not?

Replacing `formik` with `react-final-form`

React final form’s first impression from docs and readme was that it is built to be one-stop-shop for all kind of forms, which means it has a lot of built in fine grained controls for all kind of use cases. And it is async by default. Which means it runs validations async reducing any possibilities of lags due to validations.

React-final-form even has a brief dedicated guide to migrate from formik. So i don’t need to add those details. I will only add details which are specific to shopify-polaris.

So as we were using @satel/formik-polaris which binds polaris components onChange and error type properties to formik.

I could not find anything similar for react-final-form which meant i had to write my own wrappers. Which are not a big deal but its always nice to have plug-able solutions instead of writing your own.

Here is a gisti created with code for the wrappers/adapters and notes on their usage.

Using Yup validation schema with `react-final-form`

There is apparently no offical way to use a validation schema in react-final-form while formik had this support. I found a function somewhere in an issue on github. And that worked flawlessly for us, here is the final form of that function we used:

import { get, set } from 'lodash-es'

// For extracting errors per field for formik
export const convertYupErrorsToFieldErrors = (yupErrors) => {
  return yupErrors.inner.reduce((errors, { path, message }) => {
    if (errors.hasOwnProperty(path)) {
      set(errors, path, get(errors, path) + ' ' + message);
    } else {
      set(errors, path, message);
    }
    return errors;
  }, {});
}

 // And to use yup schema for validation:
 export const finalFormYupValidator = async (values, schema) => {
  try {
    await schema.validate(values, { abortEarly: false });
  } catch (errors) {
    return convertYupErrorsToFieldErrors(errors);
  }
}

And to use this:

import { finalFormYupValidator } from '../../helpers'

...
...

<Form
      initialValues={initialValues}
      validate={async (values) => await finalFormYupValidator(values, ValidationSchema)}
      onSubmit={async (values) => await submit(values, alert) }

Obviously you may tune above according to your needs.

Fine tuning `react-final-form` for our usage

As soon as we made switch to react-final form, we saw immediate effect of at-least 4-5x speed improvement , we could sense a little bit of lag still but it was a lot better already.

We decided to fix this lag too, so we explored other options. As our form was considerably big, we knew validations are what is causing this remaining lag. So we enabled validateOnBlur option(by passing it as a prop to Form and voila! Our form was as fast as it could get with no lag at all.

Migrating from react-native-router-flux to react-router for web compatibility

Zia Ul Rehman — Sun, 25 Aug 2019 18:10:44 +0000

Part of our effort to compile our react-native app for web, so we can have kind of a PWA built in react-native which runs on iOS, android and app, all 3 with same codebase.

We did this migration keeping the interface as close to react-native-router-flux as we could so there are minimal required changes in the app , and in some cases we were able to just change the import and that was it.

Trying to avoid replacement

We were usingreact-native-router-flux v4.0 in our RN(0.59.9) app, this version is based on react-navigation v2. When compiling for web, we obviously need compatible libraries. Although react-navigation didn’t support web officially in v2, we gave it a try, many things were working, we could navigate just fine. but issue was, there was no url paths being changed while navigating(among some other issues).

We need to show urls to users so users can bookmark, share or do whatever they want with those urls.

Let’s try an upgrade?

react-navigation v3 is officially supporting web now, although in early stages, there have been a lot of work for web support and efforts are still underway, but progress on the web part seems a bit slow(at the time of writing) with lot of important use-ability related issues open with little to no efforts to resolve those issues. But as it was most easy to just try upgrading our existing library’s version, we gave it a try.

react-native-router-flux v4.1(which is in beta, at the time of writing) uses v3 of react-navigation, so we upgraded to that and issues were still there. Issues like:

No URL paths, only domain name kept showing on all pages.
Pages were not scrollable on web when bottom navigation was in use.

So as odds were heavily against this setup to work, we decided to just change the library and choose an alternative.

Why `react-router`?

react-router is the most mature library which supports both native and web , although packages for web and native are different but API is almost identical, which is a huge benefit, as it allows us to conditionally import(based on OS) and forget about the rest, as API is same(you will see this implemented in a minute).

It is also a good choice for cross platform compatibility because unlike react-navigation it only concerns itself with navigation, building UI for bottom or top or whatever navigations is not its job, that reduces parts which needs to be fixed for web and makes compatibility a lot easier.

Gotchas

Before we jump into conversion, lets discuss some gotchas:

Depending upon size of your application, this can be a lot of rework, we did worked almost 2 weeks to set our app fully working with this new library.
You can’t pass functions(callback etc) or any other serializable object in state for next route.
You will have to redo any non-routing features you are using from react-navigation (i am referring to react-navigation as react-native-router-flux is based on that and all features it gives us are based on those in react-navigation). What I mean by non-routing features are features like building UI for bottom navigation and anything like that.
To keep the API same, we will use render props pattern to spread initial props/state and even url params to the component being rendered on any specific route.

Actual Conversion

First, add these packages: react-router-dom and react-router-native, make sure their versions match.

Now, we will need to write a couple of special files, which will contain platform specific conditional imports etc. Then, we will use these files to import related components instead of importing from actual libraries. You can consider it kind of wrappers.

We will be taking an incremental approach, so we will not remove the router-flux in start, rather we will keep it until we are finished with our whole process.

Let’s start with magic files(you can copy these as is).

Magic Files

1. `Router.js` file:

import { Platform } from 'react-native'; 

const isWeb = Platform.OS === 'web'; 

const RouterPackage = isWeb ? require('react-router-dom') : require('react-router-native'); 

export const Router = RouterPackage.Router; 
export default RouterPackage;

_ Disclaimer: This(and some others) code is taken from somewhere on the web(maybe it was a github issue or something)_.

Place this file at the location of your preference(we kept it at root, for now). This file is doing nothing fancy, it is just conditionally importing correct library, when its web, we import from react-router-dom otherwise we import from react-router-native.

2. `RouteHistory` files:

For this, we needed to create two files, first RouteHistory.js:

import { createMemoryHistory } from "history"; 
export default createMemoryHistory();

And for web another file, RouteHistory.web.js:

import { createBrowserHistory } from "history"; 
export default createBrowserHistory();

If you don’t know what this .web.js extension is, and you are not working on a project which will compile to web, you can skip the web.js file. As, keeping it short, .web.js file is used instead of .js file on the web, when available. Moving on.

What these files are doing is just creating a history object , which we will pass to react-router so a proper history strategy is used as per platform, memoryHistory on mobile and browserHistory on web.

Routes Mapping

Now we have our magic files in place, now we will start our actual conversion process.

For routes definition, i made 3 files:

1. `urls.js` file:

This file is simplest, it just contain url definitions:

export default {
    applyJob: '/job',
    browse:   '/job/:job_id'
};

2. `RouteActions.js` file:

This file will be used to navigate to different routes, it will replace import { Actions } from 'react-native-router-flux' with something like: import Actions from '../RouteActions'. And hence, will be on forefront for keeping the API same as router-flux’s Actions.

import { Platform } from 'react-native';
import history      from './RouteHistory';
import urls         from './urls.js';
import { generatePath } from "react-router";

export default class RouteActions {
  static pop() {
    setTimeout(() => {
      history.goBack();
    }, 1);
    return null;
  }

  static base(componentProps = {}) {
    history.push('', componentProps);
  }

  static post(componentProps) {
    const url = generatePath(urls.post, {postId: componentProps.postId});
    history.push(url, componentProps);
  }

  static posts(componentProps = {}) {
    history.push(urls.posts, componentProps);
  }

  static refresh(componentProps = {}) {
    history.replace(history.location.pathname, componentProps);
  }

  static replace(route, componentProps = {}) {
    history.replace(route, componentProps);
  }

  static jump(route, componentProps) {
    history.push(route, componentProps);
  }
}

Note that there are quiet a few extra routes , like pop, refresh, replace, jump and base, these routes are implemented as our app was using these routes a lot from react-native-router-flux, we tried to maintain as much interface as we could to match that of Actions from react-native-router-flux. And hence, we had to implement these functionalities.

Secondly, we are receiving componentProps and passing them in push and replace functions. This is also for the effort to match Actions behavior in react-native-router-flux, you will more details about this in Routes.js file description.

You might also have noticed that componentProps is optional in all other routes, but not in post function/route. Why?

This is because we are expecting an object will be passed, which will contain at-least postId, and can contain any other props it want to pass to the component being rendered o the route.

Take your time to try to make sense of this file. Then continue.

3. `Routes.js` file:

This file will be a replacement of old react-native-router-flux routes definitions , whether that was in a separate file or mixed up in App.js, this new file will replace those routes definitions and will actually be imported in App.js to render routes replacing router-flux’s import.

For the sake of simplicity, let’s assume there are only three routes we have to map:

/posts (is home page after signin)
/post/:post_id
empty/root route which renders home page

It will look something like:

import React, {Component} from 'react';
import Routing, { Router } from './wrappers/Router/Router';
import history from './RouteHistory';
import urls from './urls.js';

//Screens
import Posts from './screens/Posts';
import Post from './screens/Post';
import Home from './screens/Home';

const { Route, Switch, Redirect } = Routing;

class Routes extends Component {
    render() {
        return (
            <Router history={history}>
                <Switch>
                    <Route exact path='' component={Home}/>
                    <Route exact path={urls.posts}
                        component={(props) => <Posts {...props} {...props.location.state}/>}
                    />
                    <Route exact path={urls.post}
                        component={(props) => <Post {...props} {...props.location.state}/>}
                    />
                    <Redirect to="/" />
                </Switch>
            </Router>
        );
    }
}

export default Routes;

We are importing our three screens and rendering them based on the url, lets break this down further:

Switch is a component which is used to render something based on some condition, it stops looking for matches whenever it finds first match, you can read docs for more info on this.

If we are on root path, without any sub paths, it will render Home component, as you can see in the code snippet.

If we are at /posts route, it will render Posts component, but that is a bit different, it is using render props pattern here, let’s discuss why.

You can notice we are spreading a couple things while rendering component:

component={(props) => <Post {...props} {...props.location.state} {...props.match.params}/>}

This is to match the react-native-router-flux‘s interface for navigating(which is something like

Actions.post({postId: 100, someOthrProps: 'value'})

{…props} : this is spreading default props from react-router, this includes history, location and match(and maybe more). Please explore docs for details.
{…props.location.state} this will spread any values we are passing when routing with Actions.post just like react-native-router-flux does, it calls this PARAMS in its docs. So from our example Actions.post({postId: 100, someOthrProp: 'value'}), post component will receive two props postId and someOthrProp. This is same concept from router-flux, if you are here reading this doc on replacing flux, I assume you have worked on router-flux and will understand what I am talking about here.

Lets use this new router

Now we can open our App.js file and start using this router, something like this(it is simplified a lot, of-course):

import React, {Component} from 'react';
import { View } from 'react-native';
import Routes from './Routes';

export default class App extends Component {
    render() {
        return (
            <View>
                <Routes/>
            </View>
        );
    }
}

Not to mention, you need to remove react-native-router-flux‘s import and initialization part in here.

Finally, replace imports:

At this stage, all your routing will be broken, you will be going to base/home route just fine, and any links or redirections will throw errors, thats because we have removed flux initialization from App.js and still trying to route using flux(in other components), lets start fixing our imports from:

import { Actions } from 'react-native-router-flux'

To:

import Actions from '../RouteActions'; //fix path obviously

And most routes should work, given that you have mapped all your urls in all 3 files, urls.js, Routes.js and RouteActions.js.

(This 3 file structure is obviously optional.)

Lookout and workout the gotchas and incrementally check and fix all links/redirects mapping routes in appropriate files.

Are we finished?

Not if you have deeplinking or universal links in your apps, in that case a little extra work might be required. We will not look into that here. you will have to figure that out yourself.

And we are done!

Thanks for bearing with me, do let me know if some parts in this article are hard to follow_(I understand there might be, but for someone who has worked properly with react-native-router-flux, this should not be hard to understand, still I am open to suggestions for improvement.)_

Cheers!

Part 1 – Converting react native app to react-native-web (react PWA) in monorepo architecture

Zia Ul Rehman — Sun, 28 Jul 2019 18:18:33 +0000

TL;DR :

This series is about my journey of converting and existing react native app to render on web with same(90%+) code, using the brilliant react-native-web project whichcame out of twitter.

Prerequisites:

This article assumes you already have some experience with reactjs and react-native, you know how to build mobile builds of a RN codebase, and have them run in a simulator.

Disclaimer:

I am mostly experienced in Ruby on Rails full stack development with mostly focus on backend and DevOps, i have worked on and off on different JS frameworks, but I am not an expert(yet) in babel and webpack configurations etc. So proceed with caution, and point out any improvements in the process in comments.

I am not yet sure how many parts this series will take. Also my work of conversion of this app is still WIP, so who knows what and where that work ends up. But i will share my journey anyway.

Also note, we did not had any active android builds, so this series won’t be explaining steps involved with building android successfully. But you should be able to have them run in the similar way as i will explain for ios below.

You can skip Credits and background section if you like and can jump to “_Let’s Just begin the actual stuff!” section._

Credits/Shoutouts:

Before we begin our journey, i wanted to give credits to a couple of people who helped a lot on this journey, first one is the Bruno Lemos, i got the react-native monorepo setup with initial react-native-web setup from his excellent blog post. And other one is Thomas Gladdines, he was so kind to help me through email with all queries i had in the process.

Background:

On a product i am working on, they have a fully functional react-native app released in iOS app store (It did not had android builds and releases, we are planning on that as well but thats not the topic here). We had its RN version recently upgraded from 56.x to 59.9 , the app IS on app store, but with not a lot of users just yet, as it is a startup still trying to get kick-started with their initial contracts and everything. Anyway.

Suddenly one morning, we have an emergency meeting, and the project manager tells us that we have these X and Y clients whom we are going to partner with, and these are business critical deals. And the issue is, they both need a web app instead of the mobile app, that too ASAP. And we need to port our react-native app to web version within a week or so, knowing we are team of only two devs. Yup! Just like that.

And incidentally, both of us had not any experience of converting apps to web. So I was tasked to do some R&D and provide a feasible plan for this conversion with minimum friction and time requirements. So as per pressure from the business side, we had to choose something where we have minimum to none learning curve.

In a perfect world , we will just be putting our code through some code convertor which coverts our react-native app to reactjs web app. But we don’t live in a perfect world, do we? Turns out, react community is making great progress towards hybrid apps and PWAs but it is not yet quiet there, ionic has its react beta out, there is a project called ReactXP from microsoft and expo is also working on web compatibility of its apps. It almost felt like we needed this conversion a couple of years too early.

So keeping in mind our short time notice and business criticality, we just decided to keep our learning curve low and not worrying too much about future and to use react-native-web, which currently officially supports Rn 0.55, and we are on 0.59, as we had saw some people mention they are successfully running RN-web on RN 0.59, some mentioned some hacks like making react-native-web think its RN 0.55 when compiling for web. So we just decided to go with RN-web to get something running on web ASAP.

I had to do a few re-dos to make it all work, thats why i thought it worths to write about it in detail, so people are not stuck where i did stuck and had to re-setup everything in a hope it will work, kind of restart

Maintainability and future?

This is a tough question, there is active progress on RN web, although it doesn’t look very active as far as commits go, and its having hard time keeping up with latest RN versions, but as more and more people start using this, more and more people will collaborate and participate in its development, and we should see better future.

The monorepo architecture itself is quiet amazing and works flawlessly with RN(except the issues of linking), but only issue we can foresee with react-native-web is its active development. Its great to try out and get something running. But if you face too many issues and/or missing features you can’t live without, and you also happen to have lot of time to transition to ionic or ReactXP or anything like that, you can try those out. But as I mentioned, those are also not yet that mature, so good luck with that as well.

Let’s Just begin the actual stuff!

Before starting, i should stress on importance of version control, just keep committing every small step which worked with proper commit messages, and you will save a lot of time.

Ok, here we go.

Initial monorepo setup:

As i mentioned above, i am using the monorepo architecture to share code between mobile and web, and i followed an excellent blog post on the matter. There is a boilerplate repo also linked in the article, which you can use. But i preferred to setup this whole thing from scratch as described in the post i linked, so have a better sense of understanding of whats actually going on. Turns out, there is a lot going on.

BTW, there is an interesting debate of monorepo vs multi-repo architecture, we are not going that route. But it worths mentioning, when we are using something like RN-web, it makes most sense to have this architecture. Maybe not that much if we were just sharing the services and reducers etc among web and native react.

Ok, now we have our monorepo setup, with basic boilerplate running on both mobile and web. Let’s start importing our existing mobile app to this architecture. Before we do that, i must mention that you can try converting the existing app to monorepo setup in place, but that didn’t work for us, was taking too much time that we decided to just port it over.

Little walkthrough of the monorepo setup

During the setup from above guide(which you have to do from there, i will not be doing that all again over here), you would have noticed we are using yarn, and its workspaces feature. We have a folder called packages on the root, it contains currently 3 sub-folder, each with their own package.json file. but important part is, all the packages will be installed in the node_modules folder at the root. Not in sub node_modules folders.

Lets go through these packages/folders one by one:

1- Components: This folder will hold generic components or the shared code so to speak, in this component we will have everything that we want to share. You can get creative with the naming if you like.

2- Mobile: As name suggests, this folder will hold our mobile specific code. And when working/building mobile we will be staying in this folder. This folder further has the usual folders we see in any react-native app, the ios android src folders etc. You should already know what are those and how we use them.

3- Web: This is where actual web magic happens. This is supposed to be focus of this article.

Gotchas!

Just like there is no free lunch, there are a few issues I have came across. I am pasting first 3 straight from Bruno’s original article.

react-native-web supports most of the react-native API, but a few pieces are missing like Alert, Modal, RefreshControl and WebView
If you come across a dependency that doesn’t work well with the monorepo structure, you can add it to the nohoist list;
react-native link might not work well with monorepo projects without nohoist; to workaround this, use nohoist on **/react-native OR instead of installing the dependencies only using yarn workspace mobile add xxx, install them in the root directory as well: yarn add xxx -W. Now you can link it and then later remove it from the root package.json. (So far what I have been doing is the second option from this. Which is to copy dependencies to root packakeg.jsonand later remove them from there.)
Lookout for library versions when you copy the dependencies from existing app, they WILL change and may jump to latest versions without you noticing if you don’t properly lock them in package.json, so i recommend you spend some time locking them properly and than making sure everything works in old app, before starting to port it.
Be prepared to have some details of the functionality of your mobile app to be compromised, at-least for short term. As odd as it sounds, its a reality I faced, some RN components are not even supported in RN-web, one of them is Alert, which we happen to use a lot, so we will need some patch(if we can find any) to make it work or we will have to use something else to achieve the same functionality.
Unless you decide to limit the width of your web app, you will probably have to fix a lot of responsiveness issues. And for some components/screens of your app. It may even feel like re-writing the view layer. As you can imagine the drastic changes for responsiveness.
I have not yet setup some proper versioning script like the one we were using in the old setup, but that might be a bit of problem for us. And we may need something custom for that, unless we can live without this and waste time every-time we need to release a new version.
Lastly, as of point 4, 5 and 6, you can already guess, this RN and RN-web combo will work best for new react apps aiming for PWA or hybrid results, so to speak. This doesn’t mean it doesn’t worth a shot when you are short on time and want something out on web from existing RN codebase.

Importing existing app to this new setup

Let’s start with dependencies

I decided to first have dependencies moved, installed, and yarn.lock file properly updated and than copy actual code. So i did that. I copied everything under dependencies and devDependencies sections in old package.json(except react and react-native, any any other duplicates), and moved it over to packages/components/package.json sections respectively. (Note that I was not careful about the gotcha number 4 i mentioned above, that caused number of issues for us later on, so lookout for that). After i got everything installed and mobile app was still running fine(of-course, because we haven’t imported anything from these new dependencies), i made a new commit.

Now import actual code

This particular step may sound like an easy thing, but it’s actually not. Believe me, I spent at-least one and a half of day to make it build for mobile in this new architecture! Yup…

To import old code, i will explain what strategy i took, and you can decide what works best for you. I decided to copy whole existing code to components package of the app, which is for the shared code. Idea was to first build the mobile successfully in this architecture, then try on web, and move what needs to be moved to mobile or web specific sub-packages.

So i just copied everything from my old src folder to packages/components/src and in our old setup, we had App.js outside of the src, i moved that inside of the src as well, and had to update some import paths in App.js but that’s fine. Now app should work? No, don’t forget linking and other xcode specific settings your dependencies may require.

Linking libraries and `xcode` fixes for our app

There might be a better way to do this than what i am going to describe by my experience.

As mentioned in point number 3 of the gotchas section, linking is a bit tricky. Before linking though, if you use cocoa pods, please setup those. Even if you don’t, this might be a good time to use them. Install cocoa pods. When it looks good, choose how would you like to link the libraries, as described in above mentioned section, for now, what i am doing is copying over all dependencies from packages/components/package.json to main package.json, yarn install and run react-native link from main directory. Then remove these dependencies from main package.json and yarn install. Last step, obviously, inside ios directory run pod install.

I had some complications , which turned out to be caused by something else, which i will explain in a minute, but on this stage, what i did was to copy over my old podfile completely. I mean if you have a podfile, it didn’t hurt to copy old one and just fix the references to the node_modules folder present in main directory of the repo.

At this step, you can try and run your .scworkspace file of your project, clean the build have the metro bundler running in the background (as described in the monorepo setup guide i linked command is something like: yarn workspace mobile start), and let’s try building the app. If you are lucky enough, your app will run, mine didn’t. It did built successfully, but it failed when loading files from metro bundler.

Strange `path` and `fs` errors

First i had bundling failed: Error: Unable to resolve modulepath... error, i blindly added a package named something path(you can google it up, i don’t remember, maybe it was named just path). Then i started seeing bundling failed: Error: While trying to resolve modulefsfrom file... and node_modules/fs/package.json was successfully found. However, this package itself specifies a main module field that could not be resolvedthats when i thought maybe, **it is something wrong with my config** , not my dependencies, because i have all my dev and other dependencies installed same as in previous app. So it is not the dependencies i need but something else. What is turned out to be? _I missed to copy babel presets from oldbabel.config.js` to new one in mobile package. And that solved both of these issues. _

That was the part where I wasted a lot of my time. after this, i retried the build, low and behold! another error of-course

`react-native native module cannot be null` errors

This error was appearing after loading from metro bundler, turns out, it is complaining about missing proper linking. As some libraries don’t just require you to run react-native link they sometimes involve manual steps as well, like the react-native-permissions package. So i had two options at this stage:

Go through all my dependencies readme, and make sure we have everything setup as needed.
Match the old .xcworkspace file libraries and linked section and make sure we have everything matching up.

For the shortage of time, i took route 2, i don’t recommend it, especially if you have android builds active as well. This took me some time to make sure we have everything we need. And after a few rounds of failure with similar errors, i was able to run the app.

NOTE: This is the section i am almost sure about that this would have some better way of doing this than the above mentioned two, definitely better than the approach I took. Comment if you can suggest anything, and i will add it in the list for readers.

Boom! Mobile app is running in monorepo setup!

There might be some other bits according to your setup, but that was it for me and i had my mobile build running! Finally!

PS: As i mentioned, i had not properly locked my dependencies, i had some issues in the app at this point, but it was working, and our only goal was to have a web version ready ASAP. So we just took note of those issues and moved on.

Well, at-least for part one. We have our mobile app running in this architecture. We can go from here in the next part and actually start porting for web app.

Please share your experiences and anything you want to add/fix in the article.

Till next time, TC.

Part 2 can be found here.

DEV Community: Zia Ul Rehman

Story of an SQL query ~10x speed-up with a simple tweak

TL;DR

Where to start fixing bad performance

Performance monitoring tooling

After Identifying Slowest Transactions

Benchmarking your fixes

Simple SQL query tweak improved it ~10x

Original Query

Original Query Plan

Start with small and obvious things

Remove duplicate join

Further optimisations, learning where to stop

Working around un-maintained redux-token-auth for redux and react 17 upgrade

Tl;DR:

Why are we sharing this?

Full Story

Devise gem force confirmation on email change and remove old email

Problem?

Solution:

Shopify-polaris form building with Formik, React-Final-Form and Yup

Disclaimer

TL;DR

Long version

ENOUGH OF THE SHOPIFY, WHERE ARE FORMS!

Building forms with formik

Trying to fix lags while sticking with formik

Memoization

Try to reduce re-renders to 1

Replacing formik with react-final-form

Using Yup validation schema with react-final-form

Fine tuning react-final-form for our usage

Migrating from react-native-router-flux to react-router for web compatibility

Trying to avoid replacement

Let’s try an upgrade?

Why react-router?

Gotchas

Actual Conversion

Magic Files

1. Router.js file:

2. RouteHistory files:

Routes Mapping

1. urls.js file:

2. RouteActions.js file:

3. Routes.js file:

Lets use this new router

Finally, replace imports:

Are we finished?

Part 1 – Converting react native app to react-native-web (react PWA) in monorepo architecture

TL;DR :

Prerequisites:

Disclaimer:

Credits/Shoutouts:

Background:

Maintainability and future?

Let’s Just begin the actual stuff!

Initial monorepo setup:

Little walkthrough of the monorepo setup

Gotchas!

Importing existing app to this new setup

Let’s start with dependencies

Now import actual code

Linking libraries and xcode fixes for our app

Strange path and fs errors

react-native native module cannot be null errors

Boom! Mobile app is running in monorepo setup!

Replacing `formik` with `react-final-form`

Using Yup validation schema with `react-final-form`

Fine tuning `react-final-form` for our usage

Why `react-router`?

1. `Router.js` file:

2. `RouteHistory` files:

1. `urls.js` file:

2. `RouteActions.js` file:

3. `Routes.js` file:

Linking libraries and `xcode` fixes for our app

Strange `path` and `fs` errors

`react-native native module cannot be null` errors