DEV Community: meikay

How To Scale your Startup with Web Scraping

meikay — Tue, 29 Oct 2019 21:42:09 +0000

Web scraping is a phrase often heard throughout the tech industry. But what is web scraping? How would you apply it to scale your startup/business? We'll talk about the 3 steps of implementing web scraping to scale your startup/business.

Why is Data Important to Startups and Businesses?

Data gives you concrete numbers and allows companies to compare themselves to their competition and stay ahead! When utilized correctly, data reduces wasted money and time by improving processes and pinpointing where companies should focus their time and energy. It becomes especially important for marketing to consumers in order to gain insights into their preferences, behaviors and habits. As you collect more data, you would make the necessary changes to improve their conversions rates.

What is Web Scraping?

Web scraping also known as data scraping, data harvesting, or web crawlers is a way to extract data from websites in a usable and a structured format. Web scraping can be done in many different ways, such as manual data gathering, custom scripts, or web scraping tools such as ScraperAPI. It makes web scraping very simple by finding proxies, setting up headless browsers, and handling CAPTCHAs. ScraperAPI is great because you can get started with 1,000 free API calls and then switch over to their advanced plan. To get access to much more and 10% off, use the promo code SCRAPE217831 . Take a look at their documentation to start scraping! Web scraping tools are often the approach chosen by data scientists and companies due to them being fast, cost effective and highly accurate.

How Would You Use The Data From Web Scraping

1. Market Research & Analyze Data collected

You can use web scraping to compare market prices to keep your product competitive and win over customers from your competitors. Startups/businesses must keep an eye out for competitor promotions to see what they focus on when it comes to their target audience. This allows you to avoid repeating mistakes of unsuccessful startups and predict new trends in your market

2. Track online Reputation

Check what people think once they’ve tried your product. The collected data on reviews and recommendations for specific websites or social media will introduce you to what people think which is helpful because you can optimize your product to meet their needs. Gathering information on your industry, competitors, partners, and customers with thought-out business strategies and promotion campaigns is the best way to scale your startup/business.

3. Target your audience/market product

Using web scraping, you can research your target audience to analyze their preferences and behaviors to appeal to them and convince them to buy your product. You can also scrape data on advertising which allows you learn the mistakes and accomplishments of well-established businesses which allows you to focus on growth and not just stay afloat.

In conclusion, web scraping is very useful in staying ahead of your competition and with the emergence of web scraping tools such as ScraperAPI, it is easier and faster than ever before. Keep in mind that analyzing data, online reputation and marketing is a full proof way for startup/business growth.

Sinatra CRUD App

meikay — Thu, 24 Oct 2019 02:05:35 +0000

How to build a CRUD app using Sinatra and MVC?

My Homework Log app is made for users to be able to log in/signup/logout of a homework assignment as well as be able to edit and delete their homework assignments.

MVC Structure

Before diving into the project, I thought about what my MVC would look like. What type of data do I need in my models? How do I set up my routes and connect everything in my controllers? And what would all this look like in my views? Instead of building out the MVC structure from scratch, I used a gem named Corneal created by Brian Emory which scafolds the MVC structure for you. It’s simple installation makes set up a breeze!

I built a User model and a Homework_assignment model for my project.

Active Record Associations

For this project, we were required to use at least one has_many relationship on a User model and one belongs_to relationship on another model. Each user has_many homework assignments and each homework assignment belongs_to a user.

Password Encryption

Password encryption was required to protect user accounts from getting hacked. It uses a hashing algorithm to scramble and salt passwords. In order to implement this we installed a gem called bcrypt and my User’s table had a password_digest column. I then added has_secure_password to my User model and now have all the functionalities of the bcrypt gem.

Database Migrations

We’ve been using SQLite for our projects so far which is light weight and simple to use. In db/migrate I created a users table and a homework_assignments table to which I added the data that was needed to make this app usable. I ran rake db:migrate to create a schema file which is a graphical depiction of your database.

Render your View

At this point, we should test out our app adding get '/' do erb:index end to your application_contoller.rb . we should see a welcome page that states everything has been setup up successfully if you’re using Corneal.

Controllers

I built three controllers which include the application_controller, a homework_assignments_controller and a users_controller. They all contain the necessary routes to help the user sign-in/sign-out, sign-up, and edit.

Login/Signup

We use the action and method attributes in HTML to connects to the /signup and post routes in the controllers.

Profile Page

It’s time to be redirected to the user’s profile page. This happens when iterating over the user object and then set the dynamic route to print out the user name of the current user using the user.username route.

Edit

To support PATCH and DELETE method types, you would need to add Rack::MethodOverride inside of the config.ru file. Secondly, set up the input like so .

Ensure User can only edit and delete their own content

At this point, I need to search for the saved user ID in the database that matches the entered username and displays the user’s Homework assignments matching the username.

Validate User’s data input

In order for the user to be redirected after signing up, their input needs to be validated so that bad data does not get saved into the data base which will cause problems down the road. I decided to use active record for this where in the models/user.rb I added

validates :username, presence: true validates :username, uniqueness: true

My Experience at Codeland 2019

meikay — Wed, 24 Jul 2019 00:23:46 +0000

This year I decided to attend the Codeland conference which did not disappoint! I have been to other conferences and this was by far the most fun and inclusive one. Talks were filled with laughs and fun. We learned how we can apply programming to solving everyday problems and taking it further by creating apps that can save lives. We also received a lot of goodies from the conference such as shirts, mugs, stickers and a whole bunch of other things that we can show off.

The conference was great for new comers and veterans alike. I felt so comfortable telling others that I am a student and not feeling like I have to be an expert. They offered workshops for all levels which made everything less intimidating but still challenging. My workshop of choice was 'Building with accessibility in mind' by Luisa Morales. It was eye opening because it showed me that I could be doing more to make my website and projects more accessible.

I was so happy to see other Flatiron students who usually attend remote study groups with me! It was great to bond with them and talk about the next steps of our coding journey. I also had the chance to see Saron Yitbarek who is the CEO and founder of CodeNewbie and also put together this amazing conference. She is someone who I look up to for inspiration and motivation on a daily basis. I was a little starstruck.

Key Takeaways

Keep Building things that you are passionate about
We have special abilities that others don't, so use them for something bigger than yourself
Accessibility should be the standard when building apps
Don't be afraid to approach people (they're awesome!)
Keep the user in mind when building
Would definitely attend again!
Make the world a better place with programming :)

If you saw me at the conference and have any questions feel free to message me or follow me on twitter @meisuleen!

Ruby Web Scraper with Nokogiri

meikay — Fri, 05 Apr 2019 16:08:41 +0000

I was interested in remote full stack positions so I decided why not build that for my portfolio project? I built a Command Line Application that uses object oriented programming to scrape remoteok.io for their top 100 full stack developer jobs. The application is able to do a second level scrape to retrieve the job description and can be installed as a gem on your local environment. My project uses the Nokogiri gem which is an open source software library to parse HTML and XML.

I will be going through how I built my CLI Data Gem project in the following steps:

Step 1 Install Bundler gem

In your terminal type:

bundle install remote_jobs

Step 2 Add Necessary Files

I had to map out the models I will be using through out my program. I decided to name my folder within lib, remote_jobs and within that folder, I added 3 more files that contained my classes.

Step 3 Setup File Environment

Then, I required the files in remote_jobs.rb to set up my project’s environment. At this point, I had to figure out what objects to build and what their attributes were going to be.

Step 4 Add Dependecies

In my remote_jobs.gemspec file, I added the necessary dependecies such as Nokogiri, colorize, and pry. I also added information about my gem such as its name, version, description, authors and homepage.

Step 5 Suedo Code

I imagined how my CLI would behave and wrote down some steps. Then, I thought about what methods I would use to make my CLI behave that way. I also needed to find out what data I want to scrape.

Step 6 Scrape

I looked for the the data’s css selectors so that I am able to access them through their children. I decided to scrape the first 100 full stack jobs from remotok.io and iterated over each job object to grab the job name, company name, and url. On the second level scrape, I scraped the description of each job. However, instead of scraping many times and potentially being locked out of the website, I only scraped if the job the user chose does not have a description.

Problems I Ran Into & How I(We) Solved It

So inevitably, I ran into difficulty. During this project I was not allowed to reach out to technical coaches for help, so I relied on my tried and trusted friend Google, as well as reaching out to my peers for some advice! When you need to code a project from scratch, you start to realize your weak points. So I figured out that my understanding of object orientation was not as strong as I had thought. I re-watched some videos on that subject and implemented my understanding to my project.

As you can see in the caption above, I said I(we). This was meant to encourage you to ask for help when you get stuck. If you have tried googling everything and searched stack overflow and still can’t come to a solution, don’t be afraid to ask for help. Which comes to the next issue I bumped into, the data on the second scrape was not displaying the way I thought it should. When I ran my program, there was a {linebreak} everywhere I looked in the description. So I asked a fellow peer to take a look and see what he thought was causing the issue. We played around with the code in pry(debugging tool) and came up with using gsub to fix the issue. This method globally substitutes the first argument with the second argument. This allows us to replace {linebreak} with a literal line break everywhere it is found within the description.

clean_description = description.gsub("{linebreak}", "\n") puts "#{clean_description}".colorize(:blue)

All in all, I am grateful to have gone through this experience and can’t wait till my next project!