DEV Community: Stan Kukučka

วิธีทำ SEO ให้อันดับข้อมูลเด่นสุดเมื่อเทียบกับคู่แข่ง (ตั๋วเครื่องบิน)

Stan Kukučka — Sat, 13 Apr 2024 16:54:40 +0000

การวิเคราะห์ SEO ให้อันดับข้อมูลเด่นสุดเมื่อเทียบกับคู่แข่งนั้น สามารถให้ข้อมูลเชิงลึกด้านประสิทธิภาพในการค้นหาของสิ่งของคุณเปรียบเทียบกับคู่แข่ง กระบวนการนี้ก็ยังเปิดเผยแหล่งที่มาของการเข้าชมของคู่แข่งของคุณอีกด้วย นอกจากนี้ ก็ยังช่วยให้คุณกำหนดกลยุทธ์และลำดับความสำคัญของ SEO ของคุณเช่นกัน หากยังมีเวลาพอให้ทำในบางกรณี

ผลลัพธ์จากการใช้ฮีทแมปของ SERP - ตั๋วเครื่องบิน

ตรงนี้จะเป็นด้านผลที่ได้รับจากการใช้ความพยายามสักนิดในการเขียนเล่นกับโค้ด Python ปะมาณสองสามบรรทัด

มาเริ่มกันเลย

ในบทความนี้ ผมจะทำการอธิบายว่าคุณจะสามารถหาผลประโยชน์อะไรได้บ้างจากการใช้ไลบรารี Python Advertools (โดย Elias Dabbas) ในการปรับปรุงความสามารถในการวิเคราะห์เชิงและวิชวลไลซ์กระบวนการแข่งขันของคุณได้อย่างไร นอกจากนี้ คุณจะได้รับรางวัลจากการใช้ฮีทแมป SERP แบบโต้ตอบได้ ที่จะทำการแสดงสิ่งที่ถูกครอบคลุมต่างๆให้เห็น รวมทั้งลักษณะและอันดับโดยเฉลี่ยของคู่แข่งที่คุณต้องการศึกษา เรามาเริ่มเซ็ตสภาพแวดล้อมกันเลยดีกว่า

วิธีการติดตั้ง Jupyter Lab

สำหรับบทเรียนนี้ ผมจะเน้นไปด้านการตั้งค่าสภาพต่างๆให้เหมาะสมในการนำมาใช้กับ OSX โดยที่ผมจะเป็นคนเลือกสภาพแวดล้อมและพื้นที่ทำงานของ Jupyter Lab มาใช้ คุณสามารถข้ามขั้นตอนนี้ แล้วตรงไปที่ส่วนการเขียนโค้ดไปได้เลย หากว่าคุณคุ้นเคยกับ Google Colab อยู่แล้ว

มาเริ่มต้นด้วยการติดตั้งแบบง่ายๆลงบน Terminal ของคุณ ตอนนี้แค่ไปที่ Applications > Utilities แล้วมองหา Terminal.app ให้เปิด Terminal แล้วเพิ่มตอมมานง่ายๆนี้ลงไปเพื่อเริ่มการติดตั้ง

brew install jupyterlab

หากคุณใช้ Linux เป็นโอเอส คุณก็สามารถใช้คำสั่งเดิมนี้ หรือจะลองวิธีอื่นในการติดตั้ง Jupyter Lab บนแพลตฟอร์มที่คุณต้องการตามที่มีไว้ให้ในหน้าเพจการติดตั้งหลัก

จะเริ่มใช้ Jupyter Lab ยังไง

ทั้งหมดนี้คือทุกสิ่งที่จำเป็นได้รับการติดตั้งลงบนระบบของคุณ ตอนนี้เหลือแค่ป้อนคำสั่งนี้ลงใน Terminal

jupyter lab

ในเบราว์เซอร์เริ่มต้นของคุณจะมีการเปิดแท็บที่มี http://localhost:8888

มาเริ่มเขียนโค้ดกันเลย

ไปที่ File > New > Notebook จะมีแท็บใหม่เปิดขึ้นมาให้สำหรับคุณ สิ่งแรกที่คุณต้องทำคือติดตั้งไลบรารีก่อนในการที่จะเล่นกับตัวแปรที่จำเป็น

การติดตั้งไลบรารี
ไลบรารีเหล่านี้จะถูกติดตั้งไว้ใน Jupiter Lab โดยจะมีความแตกต่างเล็กน้อยหากเทียบกับเหล่าคำสั่ง Terminal มาตรฐานทั่วไป โดยคุณต้องเอ็นเทอร์ก่อนคำสั่ง ! (เครื่องหมายอัศเจรีย์)

!pip install advertools
!pip install adviz

ในหน้า Jupiter Lab ของคุณจะมีลักษณะเช่นนี้ หากต้องการเรียกใช้คำสั่ง คุณจะต้องกำลังเปิดหน้าฟิลด์ที่กำลังใช้งานที่มีโค้ดของคุณอยู่ แล้วกด Shift + Enter

หากต้องการป้อนโค้ดต่อไป คุณต้องกดที่ไอคอน + ตรงด้านบนซะก่อน เพื่อเพิ่มช่องว่างใหม่สำหรับป้อนซินแท็กซ์โค้ดอันใหม่

การอิมพอร์ทนำเข้าแพ็คเกจ
อะไรที่ติดตั้งไปแล้วจะต้องผ่านการอิมพอร์ทนำเข้ามาซะก่อนถึงจะใช้ได้ ดังนั้นเราจะทำการนำเข้า pandas, advertools, adviz และ plotly เข้าสู่ช่องใหม่ของโค้ดนี้

import pandas as pd
import advertools as adv
import adviz
import plotly.express as px

การตั้งค่าเครื่องมือค้นหาเซิร์จเอนจิ้นและคีย์ API โดยเฉพาะ
คุณจะต้องทำการตั้งค่า ID ของเครื่องเซิร์จเอ็นจิ้นโดยเฉพาะนี้ซะก่อน (ตรงโค้ด cx) เพื่อทำการคิวรี่เซิร์จเอ็นจิ้นที่คุณต้องการผ่าน API จากนั้นค่อยตามด้วยรายการคีย์เวิร์ดที่คุณต้องการรับผลลัพธ์มาดูหลังจากที่ทำไปแล้ว ไปที่ Programmable search engine homepageเพื่อดูว่าจะ ค้นหาอะไร? เลือก ○ ค้นหาทั้งเว็บ ในการตั้งค่าการค้นหา เลือก ○ ค้นหารูปภาพ และ ○ การค้นหาแบบปลอดภัย กดติ๊กเครื่องหมาย reCAPTCHA แล้วกดสร้าง

จากสคริปต์ทั้งหมดนี้ คุณแค่คัดลอกเฉพาะส่วนหลัง cx= นี้ก็พอ

<script async src="https://cse.google.com/cse.js?cx=YOUR-SEARCH-ENGINE-ID">
</script>
<div class="gcse-search"></div>

หากต้องการให้มี API ในการค้นหาโดยเฉพาะ ให้ไปที่ตรงนี้แล้วกด Get a Key โค้ดที่คุณได้คัดลอกไปนั้นจะอ้างอิงถึง key ในสคริปต์ สิ่งที่จะถูกส่งไปยังฟิลด์ Jupyter lab ก็คือโค้ดสุดท้ายอันนี้

cx = "YOUR-SEARCH-ENGINE-ID"
key = "YOUR-CUSTOM-SEARCH-API-KEY"

เช็คคีย์เวิร์ดด้วยการใช้ API
ในส่วนนี้ เราจะทำการคิวรี่ API ผ่านการใช้ Advertools เป็น adv โดยใช้ serp_goog เป็นตัวช่วย ผมจะสร้างตัวแปร dataz ไว้จัดเก็บผลลัพธ์ โดยคุณสามารถตั้งชื่อตัวแปรนั้นยังไงก็ได้ โดยค่า gl=["th”] จะเป็นตัวกำหนดค่าใน Google ที่คุณต้องการค้นหา ซึ่งในกรณีนี้ก็คือ Google.co.th ในส่วนโค้ดสุดท้ายของคุณนี้ ถ้าอยากคุณจะเพิ่มคีย์เวิร์ดเพิ่มเติมเข้าไปอีกก็ได้ โดยวัตถุประสงค์ของโค้ดนี้ ผมได้เลือกเพียงแค่บางส่วนเท่านั้น ถ้าคุณอยากที่จะทดลองอะไรเพิ่มเติมก็ตามแต่

dataz = adv.serp_goog(cx=cx,
                       key=key,
                       gl=["th"],
                       q=["จอง ตั๋ว เครื่องบิน",
                          "จอง ตั๋ว เครือ ง บิน",
                          "ตั๋ว เครื่อง บิน",
                          "จองตั๋ว",
                          "จองตั๋วล่วงหน้า",
                          "จองตั๋วออนไลน์",
                          "จอง ตั๋ว เครื่อง เครื่องบิน",
                          "ตั๋ว ถูก",
                          "ตั๋วเครื่องบิราคถูก",
                         ])

เมื่อคุณเรียกใช้ฟิลด์นี้ด้วย Shift + Enter กระบวนการคิวรี่ก็จะเริ่มขึ้น คุณแค่รอก็พอ

หากอยากจะรู้ว่าคุณสามารถตั้งค่าอะไรอีกบ้างในการใช้ serp_goog ให้ป้อนช่องว่างของ Jupyter Lab adv.serp_goog? (ที่มี ? เครื่องหมายคำถาม)

วาดแผนที่ฮีทแมปโดยใช้ SERP
ส่วนโค้ดตรงนี้จะตรงๆไปเลย จะให้มันเยอะกว่านี้ก็ได้ คุณก็สามารถแก้ไขผ่าน fig ค่าเกือบทั้งหมดที่คุณจะเห็นจะเป็นฮีทแมป SERP แต่สำหรับกรณีนี้ ผมจะทำให้มันง่ายที่สุดเท่าที่จะทำได้ หากเราใช้ fig.layout.title เราก็จะสามารถทำการกำหนดชื่อของแผนที่ฮีทแมปได้ ผมตั้งชื่อตัวแปรนี้ว่า dataz ผ่าน serp_heatmap ที่ตรงนี้ เราจะทำการกำหนดผลลัพธ์จำนวนหนึ่งให้เป็น num_domains (จำนวนโดเมน) ทั้งหมดนี้ คือสิ่งที่เราต้องตั้งค่าเพื่อสร้างฮีทแมปของ SERP

adviz.serp_heatmap(dataz,
                   num_domains=10)
fig = adviz.serp_heatmap(dataz)
fig.layout.title = "ผลลัพธ์จากการใช้ฮีทแมปของ SERP - ตั๋วเครื่องบิน"
fig

ผลลัพธที่โต้ตอบได้แบบฮีทแมป SERP - ตั๋วเครื่องบิน

ตามที่แสดงไว้ตรงข้างต้น ตอนนี้คุณควรจะเห็นผลลัพธ์ที่เป็นแบบฮีทแมป SERP ในฟิลด์ใหม่ของ Jupyter Lab ได้แล้ว โดยมันจะเป็นแบบอินเตอร์แรคทีฟคือสามารถโต้ตอบได้อย่างสมบูรณ์ เพราะงั้น คุณจะสามารถวางเมาส์เหนือวงกลมเพื่อดูว่าคีย์เวิร์ดใดในตำแหน่งใดที่ถูกเรียกใช้ในแต่ละโดเมน ดังที่แสดงในแผนที่ฮีทแมป SERP แบบโต้ตอบได้อันนี้

ผลลัพธ์ที่ไม่เสถียรในฮีทแมป SERP
ผลลัพธ์อาจแตกต่างกันไปขึ้นกับวันที่คุณทำการวิเคราะห์ เนื่องจากเว็บไซต์ส่วนใหญ่ผลิตเนื้อหาและอัลกอริทึมของ Google สะท้อนถึงการเปลี่ยนแปลง และคำนวณความครอบคลุมและตำแหน่งใหม่สำหรับแต่ละโดเมน/เว็บไซต์ต่างๆที่เกิดขึ้น (เว็บไซต์ข่าวอัปเดตบ่อยที่สุด) แทบจะทันที

ยกตัวอย่างเช่น ก่อนเทศกาลวันหยุด วันตรุษจีน หรือวันหยุดประจำชาติไทยอื่นๆ สามารถทำให้การจัดอันดับเว็บไซต์บางแห่งดีขึ้นมาอย่างแปลกหูแปลกตาก็ได้ เพราะเว็บนั้นอาจมีข้อเสนอโปรโมชั่นที่เกี่ยวข้องกับ (จุดหมายปลายทาง) พิเศษให้ เลยทำให้เครื่องมือค้นหาจะดำเนินการโดยรวมผลพิเศษนี้ไปด้วย

วิธีปรับเปลี่ยนผลลัพธ์ของฮีทแมป SERP ให้เป็นไปตามตามโดเมนระดับบนสุด
หากคุณต้องการจำกัดผลลัพธ์ในฮีทแมป SERP ให้มันแคบลงเหลือแค่เฉพาะโดเมนจากประเทศใดประเทศหนึ่งเท่านั้น หรือจะพูดว่าอยากจำกัดแค่ผลการค้นหาเฉพาะจากเอกสารที่มาจากประเทศใดประเทศหนึ่ง คุณอาจใช้ตัวดำเนินการบูลีนในค่าของพารามิเตอร์ cr โดยโค้ดทั้งหมดนี่มีไว้เพื่อจำกัดโดเมนที่มาจากประเทศไทยเท่านั้น คือ cr=["countryTH"] (วางไว้ตรงโค้ดที่คุณระบุว่าเป็นคีย์หลัก) จากนั้น Google Search จะทำการกำหนดประเทศของเอกสารโดยการวิเคราะห์โดเมนด้านบนสุด (TLD) จาก URL ของเอกสาร และตำแหน่งทางภูมิศาสตร์ของที่อยู่ IP ของเว็บเซิร์ฟเวอร์

ขอขอบคุณ San สำหรับงานแปลบทความนี้ และ Goh Rhy Yan สำหรับภาพปกจาก Unsplash

How to do an SEO Competitor Analysis (Presidential Elections)

Stan Kukučka — Mon, 01 Apr 2024 06:00:12 +0000

Conducting an SEO competitor analysis can provide valuable insights into your performance in search compared to your competitors. This process also uncovers the sources of traffic for your competitors. Furthermore, it helps you determine your SEO priorities and strategy if it's not too late in some cases.

Final result of SERP Heatmap - Presidential Elections in Slovakia 2024

This will be the result of your effort while playing around with a few lines of Python code.

Let's start

In this article, I'll explore how you can leverage the Python library Advertools (from Elias Dabbas) to enhance your competitive analysis visualization capabilities. Additionally, you'll be rewarded by an interactive SERP heatmap demonstrating coverage, appearance and average ranking for competitors you would like to observe. Let's start to set the environment.

How to install Jupyter Lab

For this tutorial, I focus on setting up an environment in OSX. In this case, I choose Jupyter Lab environment and working space. You can skip this if you are familiar with Google Colab and head right to the coding section.

Let's start with a simple installation in your Terminal. Now just head to Applications > Utilities and look for Terminal.app. Open the Terminal and add this simple command to start the installation.

brew install jupyterlab

If you are using Linux you can use the same command or try a different way to install Jupyter Lab on a platform you prefer as mentioned on the official installation page

How to start Jupyter Lab

All needed has been installed onto your system. Now enter this command into Terminal.

jupyter lab

In your default browser will open a tab with http://localhost:8888.

Let's start writing the code

Head to File > New > Notebook. A new tab will open for you. First thing first you need to install libraries to play around with the needed variables.

Installing libraries
These libraries will be installed into Jupiter Lab. There is a tiny difference when it comes to standard Terminal commands.
You have to enter before the command !(exclamation).

!pip install advertools
!pip install adviz

In your Jupiter Lab environment it will look like this. To run the command you stay in the active field where your code is and hit Shift + Enter.

To continue to enter the rest of the code you need to hit the + icon on the top to add a new empty field for new code syntax.

Importing packages
What has been installed needs to be imported. So we'll import pandas, advertools, adviz and plotly. Enter into the new field this code.

import pandas as pd
import advertools as adv
import adviz
import plotly.express as px

Setting up Custom search engine and API key
You have to set the Custom search engine ID (in code known as cx) to query your preferred search engine via API then later on with the list of your keywords you would like to see results for. Visit Programmable search engine homepage for What to search? select ○ Search the entire web, for Search settings select ○ Image search and ○ SafeSearch. Tick reCAPTCHA and press Create.

From the whole script, you have to copy just the part after cx=.

<script async src="https://cse.google.com/cse.js?cx=YOUR-SEARCH-ENGINE-ID">
</script>
<div class="gcse-search"></div>

To have Custom Search API head here and press Get a Key. The code you'll copy is referred to key in the script. What goes to the Jupyter lab field is in final this code.

cx = "YOUR-SEARCH-ENGINE-ID"
key = "YOUR-CUSTOM-SEARCH-API-KEY"

Check keywords via API
In this part, we are going to query API via Advertools as adv with the help of serp_goog. I created the variable dataz where results will be stored. You can name it whatever you want. The value gl=["sk"] defines what Google you want to query. In this case, it is Google.sk.

In your final code, you can add more keywords. For code purposes, I choose just a few. In my final code, I entered quite a lot of variations of keywords and presidential candidates' names. Feel free to experiment.

dataz = adv.serp_goog(cx=cx,
                    key=key,
                    gl=["sk"],
                    q=["prezidentské voľby preukaz",
                       "prezidetske volby preukaz",
                       "prezidentské voľby voličský preukaz",
                       "prezidentské voľby kandidáti",
                       "voľby 2024",
                       "prezidentské voľby",
                       "prezidentské voľby slovensko",
                       "prezidentské voľby 2024",
                       "prezidentské voľby slovensko 2024",     
                      ])

When you run this field with Shift + Enter the process of querying will start. You have to wait.

To know what else you can set up for serp_goog enter an empty field of Jupyter Lab adv.serp_goog? (with ? question mark)

Draw SERP Heatmap
This code is straightforward. It can be more robust and you can modify via fig almost all values you will see in the SERP Heatmap, but for this case, I kept it as simple as it can be. With fig.layout.title we define title of the heatmap. I call variable dataz via serp_heatmap. Here we define a number of results as num_domains (number of domains). That's all that we have to set up to draw the SERP Heatmap.

adviz.serp_heatmap(dataz,
                   num_domains=10)
fig = adviz.serp_heatmap(dataz)
fig.layout.title = "SERP Heatmap - Presidential Elections 2024 in Google.sk"
fig

Interactive result of SERP Heatmap - Presidential Elections

As shown at the beginning you should now see the result of the SERP Heatmap in the new field of Jupyter Lab. It's fully interactive, so you can hover over the circles to see what keywords on what position have been triggered for each domain as shown on this Interactive SERP Heatmap.

Unstable results of SERP Heatmap

Results can vary based on the date you run your analysis because most of the websites produce content and Google algorithms reflect changes and recalculate coverage and position for each domain/website (mostly frequently updated news websites) almost simultaneously. For example, before the first round of the Presidential election in Slovakia, there were lots of results from domains like Wikipedia, Twitter, Facebook or Instagram mostly for candidates names.

How to modify SERP Heatmap results based on top-level domain

If you want to narrow your results in the SERP Heatmap just to domains from a specific country in other words restrict search results to documents originating in a particular country, you may use boolean operators in the cr parameter's value. The whole code for domains originating just from Slovakia is cr=["countrySK"] (place it in the code where you specify keywords). Google Search then determines the country of a document by analyzing the top-level domain (TLD) of the document's URL, and the geographic location of the Web server's IP address.

Thanks to Steven Lelham for the cover image from Unsplash.

Hosting Checker - How to find out where the website is hosted via Terminal?

Stan Kukučka — Wed, 03 May 2023 20:40:08 +0000

It can be some basic background check or a serious need to identify what hosting is used for certain web services. It can be practical to identify why is a certain website so quick to load or try to decide what hosting service will be best for a similar online service you have found.

There is tons of online service that claim to detect the hosting service of a particular website. Mostly named "hosting checkers". But you don't have certainty if the returned result is valid.

Where is your or other website hosted?

To this question, you can have a few different answers. Hosting is mostly a collection of a variety of different services. It is possible to have all your hosting services "made" in a single company. But it's not uncommon to see a having website and each of its hosting services provided through several hosting companies. As an example, services can vary to these:

Domain Registration
Hosted DNS
Web hosting
CDN
Email Hosting

In this short article, we are going to focus on the website's web hosting detection via terminal command prompt.

Website hosting checker background

This quick guide will provide you with instructions on how you can run a web hosting check with a simple command. Before a needs to be done brief installation procedure and then you can jump on research to find out who is hosting the website you would like to know about.You have to do these:

Install Nmap the Network Mapper - Free Security Scanner
Enter specific command into Terminal

This quick tut will focus on the installation process for OSX systems.

How to install Nmap the Network Mapper

Nmap binaries for Apple macOS (x86-64) are available in this Mac OS X Binaries section. You can find out the latest stable release installer is nmap-7.93.dmg. Download it and open the dmg file to start the installation.

You'll be blocked by default in your system. So you have to head to System Settings > Privacy & Security and scroll down to the Security section where you can notice this message to approve with the Open Anyway button.

Let's start with hosting checker in your Terminal

Now just head to Applications > Utilities and look for Terminal.app. Open the Terminal and add this simple command that starts with nmap then you enter the domain name without HTTP/S and WWW prefixes and at the end you run nmap script --script whois-ip. The whole command looks like hereunder.



nmap yourwebsite.com --script whois-ip

You will get a long list of open ports where at the end will pops-up these Host script results like in this example.



Host script results:
| whois-ip: Record found at whois.arin.net
| netrange: 75.1.0.0 - 75.1.134.255
| netname: AMAZO-2
| orgname: Amazon.com, Inc.
| orgid: AMAZO-2
| country: US stateprov: WA
| orgtechname: Amazon EC2 Network Operations
|_orgtechemail: amzn-noc-contact@amazon.com

Based on the netname or orgname you can easily identify what hosting is your desired website on. To be more specific in a technocratic way netrange will provide a range of IPs.

And that's all. The whole magic. How to check website hosting via the terminal anytime you need. No need to rely on random online tools with uncertain results.

Thanks to Taylor Vick for the cover image from Unsplash.

Install and create virtualenv via terminal

Stan Kukučka — Sat, 12 Mar 2022 19:33:30 +0000

What virtual environment is? Let's jump into this topic. I'll try to explain it as simplistic as possible with some comparison and plain explanatory style.

Imagine your system is some sort of space you are executing your programs. It's a space where you run some sort of other stuff you would like to work with. There are plenty of other things you can run from a developer's perspective. Each of these things can be installed on your system. But what will happen when you'll install each of this dev stuff on your system. Yes, that can be a quite big mass I would say. And in some cases, there can happen even version conflict someday.

So to avoid this there is a simple solution. That's why the virtual environment comes to play. As naming sayin' it will create some sort of virtual box for dev stuff you would like to install. It can be anything, python3, Django or your next flask project with a specific setup.

Note: If you have already installed some dev tools you can simply skip the "Install Xcode with Homebrew" section for sure cos there is a high probability you have it on your system.

Install Xcode with Homebrew

This step is just easy simple copy-paste work. Simply paste this "looong" command from brew.sh website and hit enter to execute.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Some stuff will flick in front of you in the terminal and then you'll be asked for the next step.

Press RETURN to continue or any other key to abort

After this, you'll be asked for a system password to enter into the terminal to continue. At this stage be patient. There will be a quite long time to download all stuff and you'll be notified about it with this simple info text.

Downloading Command Line Tools for Xcode

Then you'll be notified with this pop-up tiny window to install some other Command-Line for Xcode.

Installing Command Line Tools for Xcode

And then after all process when it's done you'll see this.

Done with Command Line Tools for Xcode
Done.

And be asked for the password again. Then after waiting please for Homebrew's instalment of its core stuff.

The last thing you'll be notified of is this terminal output. And it's all done. I hardly recommend restarting the whole system cos I have recognized virtual environment installation issues without having passed such.

==> Next steps:
- Run `brew help` to get started
- Further documentation: 
    https://docs.brew.sh

Install Virtual Environtment

This is simple straight forward. Just execute pip3 install virtualenv with sudo command.

sudo pip3 install virtualenv

Create First Environment

Let's start to create the first general main directory with the name "virtualenvs" and enter it in the directory.

mkdir virtualenvs
cd virtualenvs

In this example, I'll create a virtual environment for Flask which is simply a Python-based system for quick web app development. But it can be anything else you would like to have in a separate work environment and you can name it whatever you want to.

Because of this purpose, we going to install the Flask directory name will be the same.

virtualenv flask

This will create flask directory as space for virtual environment. We are going to activate this virtual environment for the further installation process. In other words, we will activate this environment to be able to install inside directory other python dependencies and libraries for further steps.

In this case for activation of the environment just type this command.

source flask/bin/activate

You'll recognize your command prompt starting with (flask) it means your virtual environment has been activated.

To deactivate virtual environment just type deactivate.

deactivate

At this point, tut could end. Activated flask environment is at its final. And now in the next step, you can install Flask, Python, Scrapy or whatever you would like to into your separated virtual environment.

Thanks to Michał Parzuchowski for the cover image from Unsplash.

Google Trends with Python 📊

Stan Kukučka — Sat, 20 Nov 2021 21:26:31 +0000

Thinking these days how we probably started to value more building blocks of our existence. As it started to feel more integrated into our existence as follows - health, water, food and often mentioned Covid that limited our ability to travel for some regions. To have overall knowledge of how does it goes worldwide there is well known Google Trends online tool to work with. But there is something special for those who are in love with Python. Beautiful graphing library Plotly licensed under the MIT license.

Scripts to Download

Here is a simple installation prepared to work with on my GitHub repository with a requirements.txt file and a short script to plot data into a nice histogram data form. Simply download these two files as follows so save these to your desktop into some working directory google-trends-analysis.py and requirements.txt Create a directory named google-trends-analysis and save these files into.

Activate Virtualenv

I do assume you already have virtualenv installed, but if not just execute sudo pip3 install virtualenv. Enter into google-trends-analysis directory with cd google-trends-analysis.

cd google-trends-analysis

To make all stuff properly done activate virtualenv with the creation of working virtual environment directory called venv.

virtualenv venv

Your google-trends-analysis directory looks like now like on picture hereunder.

Now activate this working environment.

source venv/bin/activate

Pytrends and Plotly Libraries to Install

All libraries needed to run Google Trends and Plotly correctly are saved in requirements.txt file. Now execute installation in a virtual environment we already activated.

pip3 install -r requirements.txt

This will take a while to pass all installations. You'll see in your terminal how all libraries are installed to properly run the script.

After installation to run google-trends-analysis.py script hereunder just type this command and enjoy response in your browser IP 127.0.0.1 on a random port.

python3 google-trends-analysis.py

This is how it will look like, a nicely plotted graph in the final result. (Graph images by author)

Thanks to Edward Howell for the cover image from Unsplash.

Uptime of macOS system via terminal

Stan Kukučka — Sat, 13 Nov 2021 20:10:48 +0000

You are logged into your system. In this case, it's macOS. You are facing just a simple question. Are you the only one who is logged in? I would like to give you an answer to this simple question with a bit of explanation. Here in upcoming lines, there will be a bit of theory background behind it.

Your System Uptime

In case your system is healthy (understand as an official system with applied updates), you are the only physical user logged in. You can check this logged in information with a simple command you will enter into the terminal. But there is a trick in that, will explain in next lines hereunder. Open your Terminal.app you'll find out in Finder in the section Applications and Utilities. For more informations check official Terminal User Guide.

By entering uptime command you'll be informed in one line output about the current time, number of days your system is up, number of logged-in users etc.

uptime

If you'll enter this very same command into the terminal in capital form as UPTIME, you'll receive a more readable summary of the logged-in sessions by the user.

UPTIME

The same output can be given by entering w command into the terminal.

Minimum of logged-in users

Do remember you will always have a minimum of 2 logged-in users. By default when you open another terminal session it will be counted as another active user logged in. In other words, if you open four-terminal sessions, you will see 4 users logged in +1 as the default logged-in user I have written about.

How to install pip3 on mac

Stan Kukučka — Sat, 13 Nov 2021 19:54:00 +0000

What is it about

Most of the installations you are going to use during your developer processes use PIP3 especially when you dig into the programming language Python.

The reason is simple, PIP is a package manager for this language. It allows you to manage libraries and dependencies that are added to the standard library, but not just that. It helps you to install those whos are missing during your development process.

Difference between PIP and PIP3

As these are more likely to look like shortcuts to something they are. PIP is a recursive acronym that stands for PIP install packages.

PIP is understood as a soft link for a particular installer. Your operation system will use one of your Python versions depending on what exactly is known as first in the system PATH variable.

To know what it is and where it is on this particular path it simply types this command to view the system file.

nano ~/.zshrc

When you run the PIP3 command, you can be sure that this particular module or dependency will be installed in Python3.

How to install with Brew

If you installed previously most of the stuff on your mac with help of Brew then this command is for you. If not start simply by adding this command from https://brew.sh/ into the terminal prompt.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This first installation process is awkwardly long, but don't get bored. Just wait and approve by entering what does it ask for. Then just go ahead with this simple command.

brew install python3

You can check the version of your PIP3 by entering this straightforward code as -V stays for the version number to request.

pip3 -V

You'll receive a PIP3 version number with a path to the last installation process.

Installation as another way around

You can make it without using brew just simply downloading an official file from Python Packaging Authority - Pypa.io.

curl -O https://bootstrap.pypa.io/get-pip.py

And then after simply executing installation with the command mentioned hereunder.

sudo python3 get-pip.py

The installation will pass and here you are. PIP3 is on your system as final. Ready to install and roll out your new python script with help of libraries and dependencies.

Thanks to Andrew Ruiz for the cover image from Unsplash.

Face detection in Python - On your own in few lines of code

Stan Kukučka — Tue, 02 Nov 2021 21:51:22 +0000

Let's do some face detection to a picture you would love to play with. It can be even a painting from Wikipedia you have already downloaded. In this brief tutorial, I'll go step by step to provide you solution to such a recognition potential that python as a programming language and its libraries have.

Inspiration and part of the source of code comes from this jupiter notebook file. The purpose of this article was to enrich the whole process with steps I have been missing and can be helpful for the reader or tester of this detection to easily pass through.

I would like to go step by step with this installation, cos if you'll try to do this on a fresh system there will be missing quite a lot of installing libraries to have this all properly working for you. So let's start with brew.

Let's Install Brew installation environment

The easiest way how to install all that's needed is to start with Brew. It installs the stuff you need that Apple didn’t. Copy and paste a bunch of the long code from this website and execute it.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Then after just simply update all installation.

brew update

Let's Install Python3

Now we can install Python3 cos what? Cos we have brew already installed and that's why it will be super easy. The next step is to update pip into pip3 cos we have python3 installation finalised. With sudo it will ask for a system password, so just simply provide your pwd.

brew install python3
sudo pip3 install --upgrade pip

Install Environment

For the next work, we'll need a virtual environment. To understand what it is try to imagine some virtual container where all installations will go into.

pip3 install virtualenv

Create Virtualenv

We will start with directory creation to where all next installations will go to.

mkdir facevirtual
cd facevirtual
virtualenv facevirtual
source facevirtual/bin/activate

After execution of the last command, you'll activate facevirtual environment so your command prompt will look like this.

(facevirtual) username@123 facevirtual %

Install needed Libraries

Create in facevirtual directory txt file named requirements.txt and paste inside this bunch of the code.

cycler==0.10.0
kiwisolver==1.3.2
matplotlib==3.4.3
numpy==1.21.2
opencv-python==4.5.3.56
Pillow==8.3.2
pyparsing==2.4.7
python-dateutil==2.8.2
six==1.16.0

These are necessary to work with the script correctly and have all, that it ask for. To execute it just type into the terminal this simple command.

pip3 install -r requirements.txt

This will help you to execute the script, we will create, withou asking for additional installation.

How To Create Cascade Classifier

Now we need Haar-cascade Classifier. It's a machine-learning algorithm already trained with tons of images. We gonna use a classifier that considers the human frontal faces to recognize in our target image.

Open this link and look for haarcascade_frontalface_default.xml. Easier for you to save this click this link and press save as to download file into facevirtual directory

We need image and script

Let's download image from the internet. It will be Rembrandt's painting Syndics of the Drapers. Here is a link to the image to save this image into facevirtual directory as test.jpg.

Now create a python file. Let's name it face-recognition.py and save into this for now empty file this bunch of the code you see hereunder.

No need to edit anything in script just be knowledged that cv2.CascadeClassifier('haarcascade_frontalface_default.xml') load Cascade Classifier and it is responsible for detecting frontal faces on our source picture. The last thing to know that img = cv2.imread('test.jpg') load the image we already saved from Wikipedia to process.

import cv2 #For importing OpenCV library
import matplotlib.pyplot as plt #For plotting the image

# Read Cascade Classifier
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

# Reading source image to process, update name of image with your own
img = cv2.imread('test.jpg')

# Detect faces
faces = face_cascade.detectMultiScale(image = img, scaleFactor = 1.1, minNeighbors = 5)

# Draw bounding box around the faces
for (x, y, w, h) in faces:
      cv2.rectangle(img, (x, y), (x+w, y+h), (255, 0, 0), 1)

# Showing number of faces detected in the image
print(len(faces),"faces are detected!")

# In this step it does plotting the image with faces detected
finalimg = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
plt.figure(figsize=(12,12))
plt.imshow(finalimg) 
plt.axis("off")
plt.show()

Execute the script and see results.

In this final step just type into a terminal command to execute the script we already created and wait for a while for the result. No need to install anything else cos we already fulfil his requirements with help of requirements.txt file to install all necessary libraries.

python3 face-recognition.py

You should be rewarded with an image where all faces on the painting are marked with a blue square. In the terminal, you'll find information on how many faces has been marked so the text will be as hereunder.

6 faces are detected!

Thanks to Wikipedia for the cover image.

Google Search Console can't fetch sitemap on GitHub Pages

Stan Kukučka — Mon, 01 Nov 2021 17:26:11 +0000

You decided to build your website and you did the decision to publish your tweaked website to GitHub pages. That's cool hosting for your static website to choose from.

You do want to now see this page and other related pages indexed in Google as soon as possible. So you verified your page with Google Search Console with one of the options it offers. It is uploading an HTML file to your GitHub account right next to other website files or adding a meta tag to your site's homepage.

That's great. But what if you would like to fetch the website's sitemap XML file and it won't work for you as good as you want this way?

Google search console won't fetch your sitemap XML

There has been notified some indexation delays regards to Google index speed to process new pages on its own. To beat this issue, the best way is to submit sitemap XML into Google Search Console to let know google bot about your website's structure - pages your website has. But what if after adding sitemap XML file it shows up in the submitted sitemaps section "Couldn't fetch" info?

There is no need to upload to GitHub pages file called .nojekyll as some advice has been found pointing out to this practice.

Create robots.txt file on Github Pages

To let know Google bot about your sitemap just simple create on a root level of GitHub pages file named robots.txt and write there this information.

First is about where your sitemap XML file is located and the rest of the information sayin' that your website si crawlable by any other crawlers (bots) in other words it's not blocking some specific bots from accessing content provided on a sitemap XML file as well.



Sitemap: https://your-website-url.github.io/sitemap.xml

User-agent: *
Disallow:

After you created this robots.txt file you can add a new sitemap into Google Search Console again and after apx. 12 hours will be fetched for sure.

Another way around is submitting a sitemap XML file the way to enter the whole URL string of your website with sitemap XML file at the end of https://www.google.com/ping?sitemap= like this hereunter.



https://www.google.com/ping?sitemap=<complete_url_of_sitemap>

For example:



https://www.google.com/ping?sitemap=https://your-website-url.github.io/sitemap.xml

Then enter this into your browser where you'll be notified about it as follows "Sitemap Notification Received". This will tell to Google bot to ping your sitemap XML URL and to process it.

And that's it, whole magic is at the very end of this article. Next morning your sitemap XML file will be fetched as needed.

Thanks to Richy Great for the cover image from Unsplash.

How Does Your Search Query Transit to Search Engine 🔍

Stan Kukučka — Sat, 26 Dec 2020 20:35:53 +0000

This might be a spooky topic to write about, but I kept this topic in my drafts for a quite long, long, long time.

I don't want to pick up some movie examples that many of us have seen about the internet and privacy. Privacy is these days quite a hot topic not just regards to mainstream social media, but as well to some known leaks that went public.

I have asked myself, after several tests and findings, do network itself can manipulate (even track) and modify search results of what we are looking for on our favorite search engine?

Disclaimer 1: In some parts of these topics, I'll just scratch the surface, to do not write in technical jargon as much.

The first part of the article will cover the basic theory of how does keyword search and autosuggest works and gently touches privacy setup when you first time run your favorite search engine.

While the second part will be talking about the technical aspect of the internet network you are using.

This article and topic itself will be of most interest to those readers coming in with interest of SEO and internet network. The ideal reader will have experience with technical SEO and terminal commands.

Search Engine "Magic"

You may wonder how the search engine is giving you ideas as you type your questions into the search input field. Search engine collect overall searches of search queries on a frequent basis then calculates similarity as homonyms and plurality to offer you then potential query suggestions you may then feel it is some sort of magic, in other words, he tries to whisper you what to type. Behind this magic, there is sophisticated calculation into what we are not going to dig in this article for now.

Tracking and collecting

It's one of the many technical prove that search engines track and collect your search queries and store them for other internal processes then after. A number of queries are a trigger to calculate and to display more popular queries higher in the list of suggested queries and this same query count can correspond to the popularity of the query itself.

It depends on the search engine you are going to choose from, but mostly when you launch the browser for the first time and type your favorite search engine, you'll be notified about privacy options for search activity within products that certain search engine can cover. This privacy setting can vary and differs from search engine to search engine and do not always touch just cookies setup.

A calculations to offer results

These queries the user insert into the input field of a search engine (A) then after input try to provide to the user result that corresponds to search query as the product in a product database to yield from (B) as shown on the picture below.

Product in the case of the search engine (Google, Bing, Duckduckgo, Baidu or other) is represented as search engine results page (list of websites) in case of e-shop search it can be a list of products (Amazon, eBay). A more descriptive process can be found in patent explanation number US10832310B2.

In the case of search engines, this website search result page calculation is more likely to be sophisticated (is not explained in this patent mentioned above) than in the case of common e-shops nearby, you are using to buy from.

How Does Your Search Query Travel

Once the user realizes information input into the input field for searching via the device (A: mobile, tablet, laptop), then after this information is transmitted to (B) the one search server.

This search query is represented as a URL search string in the top part of your browser and travels to the search server. To see the path how does your "search query string travel" there is a command for that.

There is a simple network diagnostic tool for showing how many different devices are between you (device you are requesting from) and target (destination server).

It sending ping (ICMP - Internet control message protocol) displaying these devices - routers encountered represented as hops. There is command traceroute in Mac OSX system for your default Terminal.app or tracepath (in IPv4 format) for other Linux based systems. To see this in form of IPv6 use traceroute6 command. For windows it's tracert command.

This command can be helpful for a penetration tester when determining what devices are on a network, but it's true that this can be often blocked by routers in the network what make this trace highly inaccurate in many cases and just briefly informative.

traceroute

Command will show what does happen in reality when you look for something on your favourite search engine and you try to request search query on such a search engine.

In this example, there are six hops till you hit google server to provide your answer. There can be more hops as shown in the example and sometimes you can receive a response without identification represented as these three stars * * * as this tool is not super accurate.

So in reality, this command can return you in terminal something like this example hereunder that has been modified for publication purpose.

traceroute to your-search-engine (123.123.12.12), 64 hops max, 52 byte packets
 1  123.45.6.7 (123.45.6.7)  39.080 ms  36.643 ms  31.633 ms
 2  123.45.6.8 (123.45.6.8)  66.296 ms  31.786 ms  33.905 ms
 3  ab-cd-1.efgh2.ij6-klmno.pr456.net (123.12.0.1)  58.625 ms  51.051 ms  39.540 ms
 4  ab-cd-2.efgh3.ij7-klmno.pr456.net (123.12.0.1)  43.920 ms  51.837 ms  42.051 ms
 5  * * *
 6  ab-cd-4.efgh5.ij9-klmno.pr456.net (123.12.0.1)  57.162 ms  58.907 ms  64.653 ms

Would you like to know what these IPs,URLs or let's say devices are? Look for Nmap tool for example to download the latest install file regards to operation system you are running.

Then run nmap -O -sV to execute the basic Nmap scan report in your terminal. This command will run for remote operation system detection as -O part of code stays for and will try to determine the version of the service running on port as -sV part of code claim. Yes and don't forget it needs to be executed with sudo at beginning and IP address or URL you have found from traceroute output at the end. The whole command looks like this.

sudo nmap -O -sV IP-ADDRESS-OR-URL-YOU-WANT-CHECK

Can Network's Routers Track Your Search?

Disclaimer 2: I have included basic pieces of information on several commands. I have not extensively dug into penetration tests and their healthy or unhealthy variety of white or black hat techniques. I will be relying on just tiny speculation for these as well as my personal experiences with certain tests, programming language, knowledge and experience i'm deeper in.

It really depends on how these routers are set up. In other words, it's a question of network setup and infrastructure of your ISP - Internet Service Provider (of your home internet service or internet service of your mobile provider, type of logs that these services capture) as well. And a really specific topic is your own home wifi router (its firmware) or device (mobile or laptop) with specific hardware, firmware or operation system you are running on that may realize their own routers visits (hops).

To rephrase it all, your input, the search url string may travel from the device first somewhere that you don't even know, even your search intent may be secured in browser terminology.

Do these devices (routers) may track search queries? The answer will they might.

Same as search engine stores and process your search query input, these devices (based on their setup and maintainer privacy decision) may collect your search query strings, internet behaviour , without letting you know.

Do these devices (routers) influent search engine result order? The answer will be positive with some specific condition.

It is one of the many other factors that influent the calculation of search results after you execute your search query but not the main one, but can have the manipulative influence to modify your search results order for sure.

Good example, how to change path trough these devices to your search engine is to change your DNS setup or turn on VPN.

Closin' of privacy topic

There can be a positive effort in search engine's data privacy towards users that can be not just limited by the ISP provider you are using but the local infrastructure you are plugged in. As your search may be "somehow" safe while you do your own personal research, you may look for artistic inspiration, or if some technical thing is already patented, browsing all that with HTTPS turned on in your favorite browser it may be collected and results of search engine modified by network setup even before you hit your favorite search engine to provide you an answer.

Maybe it can be that simple, that each search is our personal lowball. Maybe just try typing something really special into your search engine and then going out for a walk to see what'll happen.

Sayin' hello and sending tribute to the author of 1984 book.

Raspberry Pi DNS Settings Update 🔌

Stan Kukučka — Wed, 16 Dec 2020 20:22:55 +0000

Enter Raspberry Pi with SSH.

Let's connect to Raspberry Pi over SSH with this simple command and then enter the password we keep for that whatever is your device connected over WIFI or via UTP cable.

ssh pi@raspberrypi.local

Modify DNS info

Go to a specific configuration file named dhcpcd and edit it via nano command.

sudo nano /etc/dhcpcd.conf

Look for the static domain_name_servers string that at the beginning holds the # sign. It means the whole string is commented in other words not used as a command. Remove commenting sign as # and edit whole domain name server as you wish. You can go with Google's Public DNS servers that stay for 8.8.4.4. and 8.8.8.8 or select Public DNS servers on Cloudflare that stays for 1.1.1.1. Choose just one provider to paste. So based on your preference you will edit the whole string as follows. In the end, the whole string should look like this for Cloudflare's option.

static domain_name_servers=1.1.1.1

Or like this for Google's option.

static domain_name_servers=8.8.4.4 8.8.8.8

Press CTRL + X and as you'll be asked to "Save modified buffer?" press Y and then ENTER. No need to restart the whole Raspberry Pi device, just restart the dhcpcd service.

sudo service dhcpcd restart

To check if the domain name server's update was accepted and passed correctly use this last command to execute resolv file.

sudo nano /etc/resolv.conf

You should be seeing something like # Generated by resolvconf info and under it nameserver updated informations. That's all.

Thank your lucky stars. Reading this whole DNS-awesomesauce brief tutorial.

Thanks to Taylor Vick for the cover image from Unsplash.

Connect To Your Raspberry Pi From a Mac 🧰

Stan Kukučka — Wed, 16 Dec 2020 20:17:36 +0000

Your Laptop and Raspberry Pi connected

You are close to making this happen. Your Raspberry Pi OS is already copied onto an SD card (as explained in previous tutorial) and plugged into a Raspberry Pi slot. There are a few steps to make this piece of tiny hardware work for you. So let's make this magic real, let's slice and dice it whole.

Enable SSH on Raspberry Pi

In this step, you definitely need to connect your Raspberry Pi into power with Micro-USB-B on one side with standard USB on other side and plugin at least a USB mouse and yeah, don't leave your HDMI port empty to plug in some monitor too. The whole Raspberry OS should boot into the nice and tight interface. Click on the berry icon on the very left top corner and lead your mouse to Preferences > Raspberry Pi Configuration as seen on the next screenshot.

This window will pop-up in the middle of your screen. There is a place to rename your Raspberry device in the section named as Hostname or keep it default as raspberrypi. Remember this Hostname serve device connection in connection command, so if you rename it here don't forget to rewrite in command too. Example for SSH connection where whole command to serve device is pi@raspberrypi.local, but will come to this point later.

To activate SSH click on the Interfaces on the top part of the window and press the radio button Enabled where SSH info is mentioned. SSH itself will help us to connect to Raspberry via terminal when it is connected to local network via UTP cabel or via wifi without monitor plugged into device later on.

Plug in Raspberry

There are two ways how to plug in your "razz" into the network. The first and the starting step as we are going to with is our first choice. Connect via UTP cable, in case we have the same port on the laptop.

Connect to Raspberry

There is no need to identify the IP address of your Raspberry Pi device. Just simply use SSH command and try to connect to raspberrypi.local with this command hereunder.



ssh pi@raspberrypi.local

You'll be notified with this command hereunder.



Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Type yes and you'll be asked for password.



pi@raspberrypi.local's password:

Default password to Raspberry device is raspberry

Now type exit

In this step we are going to get rid of the default password to make it more safer. So you'll log in again into Raspberry Pi via SSH, but with the additional command passwd to set up a new password.



ssh pi@raspberrypi.local passwd

Pass the whole process of changing the actual password into the new one.

Turn on Wifi

If you are looking for wifi activation you should be already connected to Raspberry via SSH on UTP cable. Then paste this command to activate the configuration interface in the terminal.



sudo raspi-config

This Software configuration tool will show up. Just select the first option as follows 1 System Options.

Then head to S1 Wireless LAN to setup wifi to connect to.

You'll be asked for SSID as the name of the wifi.

Enter passphrase to log into wifi.

Here will system freeze for a while then after you confirm the password just press simple FINISH in the main interface.

Connected over wifi

Now you can unplug UTP cable and let connect to your Raspberry Pi over SSH again. Your laptop and Raspberry Pi needs to be on the same wifi. You'll be notified by this informations hereunder.



Warning: Permanently added the ECDSA host key for IP address 'HERE IS YOUR MAC ADDRESS' to the list of known hosts.

And just enter the password to get into your "Razz" again. Be ready to follow this process anytime you reinstall the system of your laptop in other words connect to Raspberry Pi over UTP cable first, set up a wifi connection, and then you can easily realize an SSH connection via wifi.

Ready to roll with your "Razz"

Now is your Raspberry fully connected via wifi (or you can connect via UTP cable if you prefer). Check how long the device is up with this command.



uptime

You can disconnect from the device with exit command when you'll stop the connection, but Raspberry Pi will still keep running.



exit
Connection to raspberrypi.local closed.

Or to turn off the device entirely with a simple command.



sudo halt

To give your Raspberry time to rest for your next upcoming projects.

Thanks to Harrison Broadbent for the cover image from Unsplash.