DEV Community: Ryo

Constructing Puppeteer Environment With Vagrant And Docker

Ryo — Sun, 25 Mar 2018 12:43:18 +0000

Introduction

I constructed the environment in which Puppeteer can be executed with vagrant and docker because I did not want to install any program into my local develpment environment.

If you install Vagrant and VirtualBox, this environment will be constracted easily.

What Is Puppeteer

Puppeteer is the library that is provided by Google and useful to headlessly testing browser.

Puppeteer can do ...

Screenshot
Create PDF
Scraping
Automating Type Form Test
Testing Javascript and the function of browser with Chrome

There was the problem about Web application with javascript like React, but Puppeteer resolve it by using latest Chrome Browser. If you do not have to do cross browser test, you should use not selenium but puppeteer.

Prepareing

Installing VartualBox
Installing Vagrant
Checking my preparing repository (GitHub Repository)

Procedure

1.Git Clone

user:~$ cd AnyFolder
user:AnyFolder$ git clone git@github.com:ikeryo1182/puppeteer_tutorial.git

2.Vagrant up + ssh

user:AnyFolder$ cd puppeteer_tutorial/vagrant_puppeteer
user:vagrant_puppeteer$ vagrant up
user:vagrant_puppeteer$ vagrant ssh

In 'vagrant up', docker will be installed by provisioning.

3.Change Directory + Root

[vagrant:localhost ~]$ cd puppeteer
[vagrant:localhost puppeteer]$ sudo su

4.Setting --selinux-enabled


- OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
+ OPTIONS='--selinux-enabled=false --log-driver=journald --signature-verification=false'

5.Docker Enable and Start

[root:localhost puppeteer]$ systemctl enable docker.service
[root:localhost puppeteer]$ systemctl start docker.service

6.Check Docker Status

[root:localhost puppeteer]$ systemctl status -l docker

This is good if the message ( the folloing ) is displayed

--> Active: active (running) since xxx xxxx-xx-xx xx:xx:xx xxx; xxmin ago

7.Docker Build

[root:localhost puppeteer]$ docker build -t puppeteer

If you want to see more detail, you should check Dockerfile.

8.Docker Run

[root:localhost puppeteer]$ docker run --rm -it -v $(pwd):/opt/data-volume -w /opt/data-volume puppeteer

9.Vagrant rsync back

user:vagrant_puppeteer$ vagrant rsync-back

You can see the image results which is created by screenshot in script.js

This is japanese page.
If you do not need japanese font, you should modify Dockerfile in which Japanese Font is installed. It is executed in Docker build.

10.Modify script.js

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox'
    ]
  });
  const page = await browser.newPage();
  await page.goto('https://google.com');
  await page.screenshot({ path: 'google_top.png' });

  browser.close();
})();

Modify script.js and Scraping as you like !

user:vagrant_puppeteer$ vagrant rsync
[root:localhost puppeteer]$docker build ~~~
[root:localhost puppeteer]$docker run ~~~
user:vagrant_puppeteer$ vagrant rsync-back

Thanks

Thank you for reading

Reference:
Oparating Puppeteer On Docker Container
Manipulate Headless Chrome With Puppeteer

You can check the source in the folloing URL (JP)
puppeteer_tutorial repository

Handling Excel File with Ruby on Rails

Ryo — Mon, 18 Dec 2017 05:32:41 +0000

Hello, everyone!

Circumstance Of Excel With Ruby on Rails

When you guys handle Excel File, there are two operations.

Read Excel File
Write Excel FIle

Today, there are several gems to handle Excel File in Ruby on Rails ,e.g roo, axlsx, rubyXL. But, it does not always mean all of them are useful.
That is some of the above gems is not equipped with all operations.

For example, 'axlsx' gem can't read Excel file and 'roo' gem can't write. Therefore, I was looking for the gems that have all operations and usefulness. And then I discoverd 'rubyXL' gem which is so useful and easier to use than other gems. In the following, I will introduce the standard method to use rubyXL.

What Is Implemented With rubyXL

On condintion that you have format file, I implement the following.

Read Excel File
Write Excel File
Export Excel File

Preparing To Implement

Install rubyXL

add the following code into Gemfile.

gem 'rubyXL'

and then you should do 'bundle install'

bundle install

How To Implement With rubyXL

Read Excel File

workbook = RubyXL::Parser.parse('ExcelFilePath')

You should replace 'ExcelFilePath' with the path in which you want read file.

Acquire WorkSheet

worksheet = workbook['WorkSheetName']

As the following, you can specify worksheet by index.

worksheet = workbook[０]
worksheet = workbook.first

Acquire CellValue

cell_value = worksheet[y][x].value

You should be careful with the location of 'x' and 'y'.

Add Cell

worksheet.add_cell(y, x ,value)

Similarly, You should be careful with the location of 'x' and 'y'.

cell_value = worksheet.add_cell(y, x ,value)

As the above, you can also acquire the value of added cell.

Write Excel File

workbook.write('ExcelFilePath')

you should replace 'ExcelFilePath' with the path in which you want put written file.

Export Excel File

send_data(
      workbook.stream.read,　
      :disposition => 'attachment',
      :type => 'application/excel',
      :filename => "FileName.xlsx"
    )

If you want binary data, you can get the data in the following.

workbook.stream.read

For Copy & Paste

#Read Excel File
workbook = RubyXL::Parser.parse('ExcelFilePath')

#Acquire WorkSheet
worksheet = workbook['SheetName']

#Acquire Cell Value
cell_value = worksheet[y][x].value

#Write Excel File
workbook.write('ExcelFilePath')

#Export File
send_data(
      workbook.stream.read,　
      :disposition => 'attachment',
      :type => 'application/excel',
      :filename => "FileName.xlsx"
    )

Thank You For Reading

In this time, I show implemention of rubyXL on condintion that there is format file. If you want to change width of cell and color of text and anything, you should read Document of rubyXL.

Thank you for much.
Have a good ruby life!