DEV Community: Tharinda Dilshan Piyadasa

Running a Python script every X minutes on Windows 10

Tharinda Dilshan Piyadasa — Tue, 05 Jan 2021 08:05:14 +0000

Recently I had a requirement of implementing a web scraping tool to identify changes in a website. It was the virtual learning environment website of my university where I wanted to create a desktop notification whenever our examination results were uploaded to the website. I started by writing a Python script that checks the website and creates a notification on Windows if updates are detected. I used the Python Beautiful Soup library to parse the HTML content of the web page and the Windows 10 Toast Notifications library to generate Windows 10 toast notifications. These tools made the implementation pretty simple.

The next phase was to run the script every 10 minutes and my initial thought was to use time.sleep() while the script was running in an infinite loop. That's when I started to realize that even though this approach solves the problem, using time.sleep() does not mean that the code is executed every 10 minutes. What it does is put a 10 minute gap between executions. For a simple expendable application(like what I'm implementing) this is not an issue but for mission-critical applications or streaming applications using time.sleep() for scheduling might have adverse effects. So I decided to use Windows Task Scheduler to run my Python script every 10 minutes.

Steps to schedule a Python script using the Windows Task Scheduler

Step 1 - Create a batch file to run the Python script

Open Notepad, add the following syntax and save it with the .bat extension.



start "" "path_to_python.exe" "path_to_python_script"

Example:



start "" "C:\Python38\python.exe" "C:\Users\Tharinda\Desktop\web_scraper\main.py"

Using start makes sure the cmd is closed after the execution. The first "" is the title that should be used with start and in this case, the title is left blank.

After double-clicking on the .bat file the python script should run if the file is configured properly.

Step 2 - Create a task in Windows Task Scheduler

Open the start menu and search for Task Scheduler.

In the Actions tab select Create Task

Give a name for the task

Switch to the Triggers tab and press New to create a new trigger.

Configure the trigger to suit your needs. In my case, the trigger is scheduled to run daily repeating the task every 10 minutes.

Switch to the Actions tab and press New to create a new Action.

Under Program/Script add the batch file that was created in step 1

Add Conditions and change the Settings depending on your requirement and press OK to save the task. You can refer the Task Scheduler documentation to learn more about the Windows Task Scheduler.

That is pretty much it. Your task should run as intended.

I hope this helps. Cheers!

Domain Specific Computing Architectures

Tharinda Dilshan Piyadasa — Tue, 15 Dec 2020 05:53:52 +0000

As stated in Moore’s Law, the number of transistors that can fit onto an integrated circuit doubles roughly every 18 months. This drove the exponential growth of computing power over the decades. However, it has become much harder to keep the pace of Moore’s law as the scale of chip components shrink. The ending of Moore’s law has caused major limitations in energy efficient general purpose architectures making implementation technology and parallelism to reach their maximum potential for performance improvements. Domain Specific Architecture with domain specific hardware acceleration was introduced to address this issue and improve the performance that cannot be achieved by improving general purpose computing.

Domain Specific Architectures achieve higher efficiency compared to general purpose computing by tailoring the architecture to characteristics of the domain being addressed. Therefore, Domain Specific Architectures require more domain specific knowledge. Graphics Processing Units(GPUs), Neural Network processors used for deep learning, and processors for Software Defined Networks(SDNs) are a few of the applications of Domain Specific Computing.

The dominance of Domain Specific Architecture

To increase efficiency and achieve higher performance, Domain Specific Computing utilizes parallelism through Single Instruction Multiple Data(SIMD) technique. As it needs to fetch only one instruction stream and the ability to operate in lockstep makes SIMD more efficient than Multiple Instruction Multiple Data(MIMD) used by General Purpose Processors. Domain Specific Architecture also makes use of Very Long Instruction Word(VLIW) that works well with explicit parallel programs by performing the necessary analysis and scheduling at compile time.

Another key feature of Domain Specific Architecture is the effective use of memory bandwidth. General purpose processors use multilevel caches to increase bandwidth and hide the latency but caches have a low temporal or spatial locality when the datasets are huge and when caches work well the locality is very high thus making the cache idle most of the time. Domain Specific Architectures use a hierarchy of memories with movement controlled explicitly by the software. Therefore when the application is specific to a certain domain, as in the case of Domain Specific Computing, user controlled memories can consume less energy than caches.

Domain Specific Architecture can also achieve higher performance by eliminating unnecessary accuracy when it is adequate. When using general purpose CPUs for applications in machine learning and graphics it uses more accuracy than needed by using 32 and 64-bit integer and floating point data. Most of these applications can improve both data and computational throughput by using just 8-bit or 16-bit integers. Domain Specific Architecture can also benefit from using Domain Specific Languages that can introduce more parallelism with improved structure and memory access.

In conclusion, it can be stated that Domain Specific Architectures can achieve better performance as they are shaped to fit the needs of a specific application but it must impact significant portions of computational problems in order to acquire a remarkable runtime impact. It is also evident that computing power is the most dominating concern driving the deployment of Domain Specific Architecture based solutions.

Instagram bot implemented in python 3.7 using selenium web driver

Tharinda Dilshan Piyadasa — Tue, 08 Dec 2020 16:31:12 +0000

I implemented a simple Instagram bot that,

Find users who don't follow you back.
Bulk liker to get attention and more followers.
Find users that follow you, but you don't follow back(fans)

Check my GitHub repo to see the implementation details. The setup guide is available in the README file.

What are your thoughts on this? Any other functionalities that I should implement?

The role of Distributed Computing in Blockchain Technology

Tharinda Dilshan Piyadasa — Fri, 04 Dec 2020 12:09:38 +0000

Blockchain is a decentralized peer-to-peer network that allows data to be stored on thousands of servers following a collective trust model among these unknown peers. Blockchain is mainly used for cryptocurrencies and the initial release of the concept was with the introduction of Bitcoin. Presently the Blockchain technology is used in various other domains such as Global Trade and Commerce, Real Estate, Capital markets, and Asset Management. In Blockchain all the transactions are recorded in a distributed immutable ledger which ensures that all the transactions that are done among nodes are available at each node and the transactions cannot be deleted or edited.

The management of the distributed immutable ledger, the backbone of Blockchain is one of the main use cases of Distributed Computing in Blockchain. Distributed Computing ensures that every node in the network has a copy of the blockchain and that it is updated in real-time when a transaction takes place. Use of Distributed Computing along with the peer-to-peer architecture, blockchain also eliminates the use of a central authority to manage the network thus making the blockchain secure, transparent, and immutable.

Another key feature in the Distributed Network of Blockchain is maintaining the present state of the ledger. Once a set of transactions is validated and a block is created by a miner that block is added to the Blockchain. Even though there are many miners working on block creation the final block that is being added to the blockchain will be created by a single winning miner and all the other nodes in the blockchain should agree and synchronize with the present state of the Blockchain. This is known as the consensus protocol followed by the Blockchain and it helps the network to achieve reliability and maintain trust within the peers in the Blockchain. Distributed Computing plays a major role in maintaining this consensus protocol as every node has to communicate with its neighbor node in order to arrive at a common decision.

Blockchain is nothing but another Distributed System that heavily uses the concepts and elements of Distributed Systems and every computation that takes place in the blockchain can be stated as Distributed System Computing.

Complications that can occur when using Distributed Computing for Blockchain

The network can be vulnerable to deterring cyberattacks such as Distributed Denial of Service(DDoS) which can exhaust the resources of a computing system by sending multiple fake requests. The majority of small Blockchain networks that contain a small number of nodes are subjected to these attacks and susceptible to fraud with hackers gaining access to the majority of nodes. Large Blockchains such as Bitcoin and Ethereum manage a ‘Proof of Work’ protocol within the distributed system to overcome these issues. This is a well-defined process that improves the trust in the blockchain.
Another complication in using Distributed Computing for Blockchain is that backups should be done on each node in the network. This process requires higher computational power and a well-connected system of nodes.
Security should be applied to each node individually as there is no central authority to manage the system as a whole and the peer-to-peer architecture of the distributed system allows any node to be accessed anytime.

Thank you for reading. Hope you like it.

Centralized vs Distributed Systems in a nutshell

Tharinda Dilshan Piyadasa — Thu, 03 Dec 2020 13:56:49 +0000

Centralized Systems

Centralized systems follow a client-server architecture that is built around a single server with large computational capabilities. Less powerful nodes connected to the central server can submit their process requests to the server machine rather than performing them directly. Internet Service Providers, Application development servers, File Systems, and Organizational Networks are some use cases of centralized networks. As the computational power of the server machine is shared among the client machines in centralized systems, the performance of the system depends on the resource allocation and scheduling algorithms.

Advantages of Centralized Systems

Easy to set up and can be developed quickly because of the client-server architecture
The whole system can be managed and monitored through the central server. Hence, a Single Point of Control.
All the resources of the system(Memory, CPU power, etc.) can be accessed by the client machines by sending requests to the server.
As there is only one machine to update, the system can be updated more efficiently.
It is easy to physically secure and maintain the server.
Less cost to set up for a small system
Client nodes can be easily removed and added without affecting the overall system

Disadvantages of Centralized Systems

Longer access time for clients far from the server and improper scheduling algorithms might result in starvation.
Problems with the central server may result in a complete system breakdown. Hence, a single point of failure.
Less possibility to maintain backups. If the central server fails and there is no backup, all the data will be erased.
Availability of the entire system depends on the central server, Therefore updates to the system have to be done on the fly. This results in difficult server maintenance.

Distributed Systems

Distributed Systems follow a peer-to-peer architecture, it is a collection of independent computers interconnected via a network. Each node in a distributed system possesses enough computational power to collaborate on a task. In a distributed system users have equal access to data and user privileges can be enabled as required. Failure of independent components does not affect the overall system which results in higher availability and improved reliability. Distributed systems have evolved to address the limitations and problems faced by traditional centralized systems such as security, data storage, and privacy concerns. Internet, blockchain, SOA-based systems are some examples where large scale distributed systems are used in practice.

Advantages of Distributed Systems

There is no central point of failure in distributed systems. The system can survive even if a node fails/is removed from the system. Hence, High fault tolerance.
Distributed Systems are both horizontally and vertically scalable. Therefore it is much easier to add new servers as the load increases in the system and take machines offline when the workload is low.
Allow many users access to a common database with minimum starvation.
The workload can be shared over multiple machines as each node possesses enough processing power. This reduces the overloading of a single machine.
The high geographical spread of distributed systems reduces the latency when getting a response to a request.
It is easy to add and remove nodes from a distributed system without affecting the performance of the overall system.

Disadvantages of Distributed Systems

As there is no common clock for the whole system it is more difficult to design and debug algorithms for the system.
It is more difficult to order/schedule transactions in the system as different nodes have different latencies and performances.
As all the nodes connected to a distributed system are independent, it is difficult for all the servers to agree upon the same information(consensus).
Messages/Information can be lost in the network system because of the absence of a central server.
Due to the distribution across multiple servers troubleshooting and diagnostics are more difficult.
Limited software support for distributed systems.
More effort must be put to make the network more secure and users have to control replicated data across multiple locations.

Centralized Systems vs Distributed Systems

Centralized Systems	Distributed Systems
Low fault tolerance as the central server acts as a single point of failure.	High fault tolerance due to the absence of a single point of failure. Servers can be added/removed without affecting the overall performance.
Low maintenance costs as the system is run through a single main server. And it is easier to monitor and manage the whole system using a single main server.	High maintenance costs because each node in the system comprises varying processing power and distributed across geographical regions.
Systems are only vertically scalable. Processing power can only be added to the central server and up to a certain limit only.	Both horizontally and vertically scalable. Servers can be added and removed with varying loads.
Less reliable as if the central server crashes the system will be unavailable for all the clients.	More reliable as if one server crashes the system as a whole can still survive.
Less throughput as the single server model will become a bottleneck when keeping up with increasing client requests.	Higher throughput as the processing power is distributed throughout the system.
The overall system is less complex as the system can be managed through the central server.	The system is more complex as we need to address replica management and maintaining consistency.

Thank you for reading. I hope you will find this information useful.