DEV Community

taroyanaka
taroyanaka

Posted on

4 1

Magical shell programming tour

Hey Hackers and programers, what do you think about "BEST PROGRAMMING LANGUAGE"?
anyway, I love shell script. It's best to me.
why?
Command line editing is best interactive way.

look this site, Japanese firebase community web page, Community Member name(名前・プロフィール) and participation number of times(参加回数).
https://firebase-community.connpass.com/participation/
I want all of twitter IDs.

seq 15|xargs -P10 -n1 -INUMBER curl -O "https://firebase-community.connpass.com/participation/?page=NUMBER"
cat ./*page*|
awk '/.*回.*/ || /.*twitter.com.*/{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "^--$"|
sed '/a>$/N;s/\n/ /'|
awk '{print $( NF -1 ) " " $3}'|
awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'|
tr -d "\""|
awk '!/^[0-9]/{next}{print}'|
sort|uniq > ./get_firebase_community_twitter_user_and_join_the_number_of_times.txt
#two liners version
# seq 15|xargs -P10 -n1 -INUMBER curl -O "https://firebase-community.connpass.com/participation/?page=NUMBER"
# cat ./*page*|awk '/.*回.*/ || /.*twitter.com.*/{print}'|grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|grep -A 1 -n "twitter.com"|grep -v "^--$"|sed '/a>$/N;s/\n/ /'|awk '{print $( NF -1 ) " " $3}'|awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'|tr -d "\""|awk '!/^[0-9]/{next}{print}'|sort|uniq > ./get_firebase_community_twitter_user_and_join_the_number_of_times.txt

so, finally I write two command lines.
"seq ~" line is simple.

programming started from, this code is show all downloaded file.
cat ./page

I tried to show times and twitter ID
cat ./page|
awk '/.回./ || /.twitter.com./{print}'

It display twitter ID and participation number of times show. but lines display, accounts which no twitter community user too.(they go over 5 times but they don't have no github account, really?)
Anyway, other remove noise lines.
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js

Still remains no twitter account IDs problem.
I resolved by displaying only ID line and the line's after one lines
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"

Added seperater lines --, but I don't need this, so, remove this
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"

Still there are line number.but I thought, awk command is ignore noise columns. so,
bound two types of data(two lines)
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"|
sed '/a>$/N;s/\n/ /'

The goal seemed to horizon. awk command ignore noise columns, change the order of columns
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"|
sed '/a>$/N;s/\n/ /'|
awk '{print $( NF -1 ) " " $3}'

Remove noise characters.
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"|
sed '/a>$/N;s/\n/ /'|
awk '{print $( NF -1 ) " " $3}'|
awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'

Just remove last of "
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"|
sed '/a>$/N;s/\n/ /'|
awk '{print $( NF -1 ) " " $3}'|
awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'|
tr -d "\""

This is goal. I thought. Just in case, Duplicate Delete lines by "sort|uniq"
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"|
sed '/a>$/N;s/\n/ /'|
awk '{print $( NF -1 ) " " $3}'|
awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'|
tr -d "\""|sort|uniq

There are any row(I don't know, any incorrect lines, haha). WTF!! I stopped thinking. anyway remove this. goodbye strange lines.
cat ./page|
awk '/.回./ || /.twitter.com./{print}'|
grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|
grep -A 1 -n "twitter.com"|
grep -v "--$"|
sed '/a>$/N;s/\n/ /'|
awk '{print $( NF -1 ) " " $3}'|
awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'|
tr -d "\""|
awk '!/[0-9]/{next}{print}'

Just in case, sort|uniq and finally Output to a file
cat ./page|awk '/.回./ || /.twitter.com./{print}'|grep -v -e twitter.com/share -v -e 参加回数 -v -e widgets.js|grep -A 1 -n "twitter.com"|grep -v "--$"|sed '/a>$/N;s/\n/ /'|awk '{print $( NF -1 ) " " $3}'|awk '{gsub(/\//," ");gsub(/class=\"event\">/,"");print $1" "$NF}'|tr -d "\""|awk '!/[0-9]/{next}{print}'|sort|uniq > ./get_firebase_community_twitter_user_and_join_the_number_of_times.txt

I got a result!!

0 1coin178
0 2sin30cos30
0 35ki
0 38elements
0 432daiki
0 84d010m08
0 92thunder
0 ANNRabit
0 AknYk416
0 AndWave
0 AndooBomber
0 Baki33
0 Caps_Rock
0 DafukuCom
0 Daik1Mochizuk1
0 Fujiyama_Yuta
0 Fumiya_Kume
0 Ike0630
0 JHashimoto1975
0 KaazTech
0 KanSAKAMOTO
0 Katsumi1972
0 KazumaNishihata
0 Kentaro0919
0 Korenari_D
0 LunwuXiaoxia
0 MHTcode_dack
0 MegaBlackLabel
0 Mockun_JPN
0 MushiAkira
0 Nagahama58A
0 NagaiSho
0 NoiG1120
0 Nunnally_Engr
0 PM_Coffee
0 SW20_Toshi
0 Sakunyo
0 Strange_bird496
0 Takaaki_Ichijo
0 Udomomo
0 UsamaMinegishi
0 VOLT3103
0 __mnc90
0 _gaharasan
0 _k0mach1
0 _kuri_megane_
0 _mochi_fuwa
0 _mono
0 _nabeen
0 _pochio_
0 _rmakiyama
0 _speech_tech_
0 _tkato_
0 a_aryzae
0 abe00makoto
0 adwd118
0 agektmr
0 akikazu_sudou
0 akitoshiyano
0 amasho
0 anotherbrain
0 asashin227
0 ats337
0 atsuya046
0 azihsoyn
0 bati11_
0 berukokoko
0 bird_tummy
0 bj1024
0 burning0204
0 c5meru
0 c7w6hgwy
0 c_keita
0 cakecatz
0 chalkygames123
0 charles_green
0 chidakiyo
0 chocodaihuku32
0 chocopie116
0 chooblarin
0 chun_ryo
0 clown_kage
0 cndtmc
0 codenote_net
0 corevale_com
0 cutmail
0 cxkpx595
0 cyberspacefarm
0 daisuke0131
0 delmontz_dgl
0 devhishiki
0 dog_jp
0 doramusukotake
0 dshukertjr
0 dskst9
0 eaglesakura
0 echizenya_yota
0 elmore_mk2
0 ema_hiro
0 f_sugar
0 falcon0328
0 fbkclanna_pub
0 feb19
0 flyocy
0 furugen098
0 g0dgarden
0 gami
0 gami_san
0 ganezasan
0 georgesekkie
0 glico800
0 gohh56
0 hamazaki_f
0 harashunta
0 hasehaseh
0 hatyuki
0 hayatoy82
0 heki1224
0 hero_mo
0 hideji
0 highballegg
0 hirokazuoza
0 hiromb1804
0 hironen
0 hiyo_ac
0 hondash918
0 hoyo1111
0 hrnkoji
0 hyshyz
0 ibihc929
0 ikawashima41
0 ikomiki
0 ikt2y
0 inouetakuya
0 isaoeka
0 isshiki
0 itinao
0 itsuyan
0 jangom2ok
0 jikkojikkonist
0 johnny__kei
0 jollyjoester
0 jumpyoshim
0 k_kukita
0 kanbo0605
0 kashi4th
0 katoma586
0 katsuhisa__
0 kaznishi1246
0 kazu19861008_gm
0 kazuki_ste6a
0 kazumasamatsumo
0 kazuyukimiyake
0 kbn1053
0 kboy_silvergym
0 ken80828086
0 kenichiro246
0 kiimiiis
0 kiyopikko
0 knt_mr
0 kochoru_kck
0 kojira
0 kokushi_yayoi
0 kotokoto_fire
0 kotonohana
0 kuluna
0 kumakumakkk
0 kunit
0 kuwahara_jsri
0 kuwapp_dev
0 kyamanaka0804
0 kyo_nanba
0 kyokutyo
0 kysnm
0 m4kvn
0 m__ike_
0 m_orishi
0 m_yukio
0 macs_6
0 maidol_28
0 mainyaa
0 maroqbee
0 masaibar
0 masakasuno1
0 massa142
0 mego_
0 mfks17
0 minhasu_k8
0 mini_will_j
0 mitsuaki1229
0 mituoh
0 miu_suke
0 miyaviee
0 mocamocaland
0 mokokko
0 moonlight_aska
0 mori_dev
0 morizotter
0 morugu
0 moruxa
0 mshrrks
0 mssknd
0 mutachii
0 n0bisuke
0 n494
0 n_atmark
0 nafu003
0 nagamejun
0 nagisa292
0 nakasho_dev
0 naokinkfj
0 naotaro0123
0 naoto_gohko
0 nashirox
0 nasneg
0 neet244
0 negi111111
0 neptacox
0 new_runnable
0 ngtknt
0 niwaken13
0 nkano_t
0 nksm
0 nomelsan
0 noppefoxwolf
0 okunokentaro
0 okyunnura
0 operandoOS
0 osada
0 ottijp
0 p_on_ro
0 pekatwo
0 pekenishi
0 pine613
0 plumsa_techbook
0 po3rin
0 prac1967
0 prototechno
0 ptiringo
0 pzt
0 qiirin3
0 r3yamauchi
0 realeyez77
0 rkanda0430
0 runble1
0 ryo_kosuge
0 sano_shoichi
0 satorunakamatsu
0 satoshicano
0 satoshoco
0 satouch
0 seo_kyou
0 shgtkshruch
0 shi_no_oit
0 shin1rok
0 shinkoizumi0033
0 shion_engineer
0 shiozaki_study
0 shirokuro331
0 shishito_megane
0 shoaooki
0 shoito
0 showkoba
0 simonTokyo
0 simorgh3196
0 sinemon30
0 sn_f
0 sota1235
0 sottar_
0 spinning13
0 sssinsi
0 stepan_ve
0 studioTeaTwo
0 suer
0 swiz_ard
0 syauta1142
0 syyama_net
0 t4ku
0 tac28
0 tadashimn
0 taisho6339
0 takahashim
0 takanakahiko
0 takanoripe
0 takuchantuyoshi
0 tanakasan2525
0 taro_osg
0 tatsuroro
0 taxiyoshida
0 tebasakyu
0 tenkoma
0 tetsuhama
0 threetreeslight
0 tikidunpon
0 tjun1
0 tmknom
0 tnagao3000
0 tommykw
0 tomo_thumb
0 tomoemon
0 tora470
0 toriimiyukki
0 toshi0607
0 tostring
0 tsh_kt
0 tshiyk
0 tsuyoshi84
0 tudumi
0 tyotsu14
0 ujikawa1026
0 ukaji3
0 uknmr
0 undbn
0 urouro_n
0 uta_tti
0 uto_ao
0 vocobeat
0 wara_be
0 webgyo
0 wkurokishiw
0 xinsuzuki
0 y_mochimaru
0 y_taka_23
0 yamatatsu193
0 yamoridon
0 yazashin
0 yokohane
0 yoneapp
0 yoshiro_m
0 youtoy
0 ysaito8015
0 yt_hizi
0 yu_webmemo
0 yuji_tsukuba
0 yuki3711
0 yuki_ycino
0 yukimarume87
0 yukotan
0 yukpiz
0 yura_yura_
0 yuta24
0 yutakiyama
0 yutam
0 yuu_ito
0 yzono
0 zaki50
0 zanyou
0 zima_gen
0 zukkey59
1 1
1 1ntegrale9
1 2OAmfYqL
1 4cteru
1 709kura
1 7L4QOI
1 8monolith8
1 95kugo
1 9m
1 AkiraTameto
1 Amothic
1 Cz_mirror
1 DotEarl
1 ENOMOTOQ
1 Fendo181
1 GenkiAma
1 Gerb_era
1 HrOhata
1 IIInaIII
1 Jxck_
1 Large2_
1 MD5500
1 MiyamotoTa
1 Niccari1
1 Quramy
1 TakayukiNakajo
1 TanakaMidnight
1 ToSurpass12
1 ToshiAizawa
1 TxRacing
1 U1Buzz
1 YuG1224
1 YuukiTakahasi79
1 _KentarOu
1 __yo7__
1 _bannzai_
1 _heiya_
1 _hyme_
1 _iamkuntao
1 _inductor_
1 _kanacan_
1 _minapin_
1 _mogaming
1 _naru_jpn
1 _neekoko
1 _r548
1 _risacan_
1 _rockname
1 _tanabee
1 _yoyoyousei
1 a1234224576a
1 affinity_robots
1 akatsuki174
1 alc_js
1 am_nimitz3
1 anannori
1 ancestor_sark
1 anoChick
1 aonorihechima
1 banjun
1 bellx2
1 bino98
1 blackdesign
1 bonetaro
1 bonkeenu
1 brn227
1 brond777
1 bukaz54
1 businessmagazi4
1 busonx
1 camcam_lemon
1 cattaka_net
1 cch_robo
1 chanibarin
1 chillmatics
1 d0nchaaan
1 d_date
1 daidaix123
1 daikiojm
1 daitasu
1 dala00
1 daniel4790
1 denn28
1 devtmyn
1 dtakanashi
1 e_tanny25
1 earshttp
1 eggman
1 entaku19890818
1 eslar
1 f_prg
1 fnwiya
1 frostndays
1 furusin_oriver
1 fuuuuumin65
1 gentlejkov
1 ginyolith
1 hamburgerdev
1 hatoyab
1 henzai01
1 hilfiger_yu
1 hiragram
1 hiro_kiku_0920
1 hiro_suzuki
1 hirobe
1 hirokazu_miyaji
1 hiromichinomata
1 hironow
1 hirothings
1 huideyeren
1 iganin_dev
1 imaizume
1 ipokoy
1 ishiichan_dayo
1 itog
1 itoha1028
1 jat_out
1 jaxx2104
1 jshosomichi
1 k8s_iro
1 k_yokomi
1 kabukawa
1 kaizumaki
1 kakajika200x
1 kamiru78
1 kasaharu
1 kazutoyo
1 keimakai1993
1 keing312
1 keitaro_1020
1 kenji_imi
1 kettsun0123
1 kgsi
1 khitomix
1 kiepng
1 kitak
1 kobo_sta
1 kokabun
1 konn_conn
1 kosuke_mtm
1 kou_hon
1 koutalou
1 krskk1
1 ktsugita
1 kzhrk0430
1 libkazz
1 lilysweet
1 limagejane
1 line6th
1 m_kumanomi
1 m_umeneri
1 ma_kun3376
1 maccostar
1 majica_ky
1 mallowlabs
1 marukugi
1 miso_develop
1 miyatomo_38
1 mizchi
1 mochan_tk
1 mogumogu_kaeru
1 mokuo_
1 momen_666
1 monry
1 moorii_76
1 moritatsuyoshi1
1 morix1500
1 mrt370
1 mrzujpn
1 muromurooka
1 mypacecoltd
1 nabeliwo
1 nagaryu000
1 nakayamachama
1 naoto_7713
1 naoto_koide
1 natsuki_summer
1 nbot0902
1 ngs
1 ninjinkun
1 noah_n6a
1 noriakihoriuchi
1 norihiro_sirube
1 nyapoo
1 occmee
1 omatty_bot
1 optaroufe
1 otofu_square
1 ovrmrw
1 pataiji
1 pco2699
1 pitown
1 poemn_618
1 popota52
1 purunkaoru
1 qushot
1 renchild8
1 rinrinhhhh
1 ryuta_01
1 sachaos
1 saicologic
1 saihoooooooo
1 salamander_jp
1 sam_oda
1 sassymanyuichi
1 satton_maroyaka
1 seclows0
1 sekitoba
1 sh0e12uatanal3e
1 shamoshamo
1 shanonim
1 shimakaze_soft
1 shmdevelop
1 shun_kashiwa
1 shuymn
1 singo405
1 siskw
1 sokuratwi
1 solty_919
1 sousoumt
1 soyanaka21
1 st5757
1 stzn3
1 sugiken_bike
1 sun54907
1 sunny_510
1 super_manner
1 systam0721
1 szdice
1 szkn27
1 t0m0120
1 ta_dadadada
1 tai_hatake
1 takahia
1 takumisatojpn
1 taquaki_satwo
1 tatane616
1 tatsumi_rize
1 tazakeeeeeeee
1 techusic_hero
1 tenbo07
1 teriyakisan
1 tetetepu
1 teyosh
1 there_you_hide
1 tiger_tokyo
1 tipo159
1 tmk_ueks
1 tmyk110
1 to_mumura
1 to_sato3
1 tocie
1 tokuda109
1 tomato360
1 tomoya0x00
1 touchez_du_bois
1 toyokun
1 tryforth
1 tuiteru
1 tutuming
1 tyorikan
1 ucan_lab
1 ukaznil
1 umechanhika
1 unecochan
1 uturist
1 uwettie9
1 voxeking
1 vvakame
1 wanpa
1 waterlow2011
1 wilf_genya
1 wkzk
1 xenbeat
1 xsoundjs
1 xyuewtqr
1 yacchin0101
1 yakisuzu
1 yamadagenki0607
1 yasu_ja_jp
1 ykarakita
1 ylisr
1 yoropan
1 yositosi
1 yossy_yuma
1 yuki_iwnr
1 yuri_htt
1 yuru36
1 yut148
1 yutaabe200
1 yutailang0119
1 yuyuyuyuyuri
1 yyyske3
1 zabio_3
2 1amageek
2 2
2 7omich
2 Gazyu
2 Kaji_____
2 Ryo_N03
2 _naventus
2 _sgr_ksmt
2 alligator_tama
2 applideveloper
2 axross_
2 boiyaa_
2 call_me_zhizi
2 darquro
2 fromkk
2 gogoroya
2 h_sakano
2 hiratti41
2 ichigoro
2 ijin
2 ippo_012
2 isopppcom
2 itomoyasu
2 k03work
2 kaiba
2 kajitack
2 keito5656
2 kuu_dou
2 kw_yt
2 laco2net
2 locol23
2 m_h_dev
2 mah_lab
2 masashi0509
2 minyako
2 mogmet
2 oppey
2 orange73_m
2 plum_eule
2 r_fujimoto
2 sasurau4
2 sekikazu01
2 shiogen
2 steelhearts_if
2 t32k
2 takaaki_sugar
2 takamii228
2 takasek
2 takayukiokazuka
2 takehilo_kaneko
2 tokyocity0406
2 tom_nobu
2 toshihirooya
2 vvaka1122
2 whiteax1
2 wiggling
2 wknar
2 wtnVenga
2 xasm
2 yagi_suke
2 youhonda30
2 youya66Official
2 yujishimojo
2 yukin_557188
2 yutaaa_8
3 1013Youmeee
3 YANOKURO
3 anh_pp
3 atotto
3 axhill
3 c_deutinger
3 canoefishing
3 d_machi0701
3 eiskst
3 fumiyasac
3 ha_hagawa
3 himrock922
3 k2wanko
3 kahirokunn
3 kato1628
3 katsummy
3 keygx
3 konoki_nannoki
3 kosuke_tech
3 maruyama_jp
3 miyazaki_e
3 mugenou
3 nowhito
3 okarina711
3 raindrop_nao
3 ry_kgy
3 t_dooor
3 up_weekend
3 uriuriuriu
3 yama_shin0525
3 yamacraft
3 yusuke_yasuo
4 Kesin11
4 aaatoyooka
4 daidai3110
4 daneko0123
4 daylilyfield
4 grem_ito
4 hecateball
4 isatanisatan
4 kame_f_no7
4 keizi666
4 kiyoshi_ohno
4 lovemeg
4 mizu_gako
4 nhiro78
4 ogawa0071
4 panchan9
4 takaoka29
4 takehiko77
4 tarotaro4
4 ticktakclock
4 yshogo87
5 Yatima_K
5 ita_3y
5 marere_re
5 silverskyvicto
5 sinmetal
5 techsmith8
5 tohda_chitoseka
5 yh1224
6 icchi_h
6 startailall0
6 tori0v0
7 kaz080

Do you think this is sloppy?
No, I don't think that.
Writing command line is no simulation results, no variable, no loop statement, and no editer.
You get best of interactivity.
Reading shell script is sucks. I think so that.
Writing command line is great, good experience is more than REPL.

I will continue to write shell program.

Hostinger image

Get n8n VPS hosting 3x cheaper than a cloud solution

Get fast, easy, secure n8n VPS hosting from $4.99/mo at Hostinger. Automate any workflow using a pre-installed n8n application and no-code customization.

Start now

Top comments (0)

AWS Security LIVE!

Join us for AWS Security LIVE!

Discover the future of cloud security. Tune in live for trends, tips, and solutions from AWS and AWS Partners.

Learn More

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay