DEV Community: Jordan Hansen

Idaho Secretary of State Business Search

Jordan Hansen — Fri, 19 Nov 2021 15:19:41 +0000

The Idaho Secretary of State business search is a great search that has many different search options. You can search for any kind of registered business entity, such as LLC, corporations, or partnerships.

Plus, it’s where I live. Home sweet home!

Here is the official search page.

If you want a way to search for a business in any state, please feel free to use our verify a business tool here, free of charge. It looks a little like this. It allows you to also easily copy and download the data.

While it is not required to be registered with the ID Secretary of State to do smaller business before you do anything like open a business checking account or get a business loan, you must register with the state. Because it is public you can find business owner information for any business registered in the state of Idaho.

I rate it 8 out of 10

Pros

Search by registered agent
Search by filing date range
Can view filed documents
Can search by active status

Cons

One of the few states that doesn’t allow you to search by entity id
Can’t search by officer

Idaho Secretary of State Entity Search

Here it is. Of course, I searched for “pizza”. The results came back very fast at .160 seconds. Here’s what the results look like:

All the normal, helpful stuff like Business name, status, filing date, and agent. I’m not sure the different between “Active-Current” and “Active-Good Standing”. Both of those sound good.

Selecting a business opens a side panel with the details of the business.

It doesn’t list officer information but other than that all relevant information is there.

Now, let’s try the agent search.

Okay, it’s the same. The registered agent won’t always be the officer but it’s kind of close if you’re looking to search by owner.

Document information looks like this:

If you were desperate to find officer information you could get it from the “Initial Filing” or “Annual Report”. Annual reports look like they track owner change.

And that’s Idaho’s Secretary of State business search.

Extra Credit

For all of my power users out there. You can access any of this business data via API. This can be handy for things like:

Business verification
KYC/KYB (know your business) anti-money laundering
Business credit check
Finding business owner information

It’s as simple as this:

let url = `https://apigateway.cobaltintelligence.com/v1/search?searchQuery=${encodeURIComponent(businessName)}&state=${state}`;

const axiosResponse = await axios.get(url, {
    headers: {
        'x-api-key': this.apiKey
    }
});

return axiosResponse.data;

Get a free Secretary of State API key here.

For more technical Idaho scraping tutorials, check it:

The post Idaho Secretary of State Business Search appeared first on Cobalt Intelligence.

Indiana Secretary of State Business Search

Jordan Hansen — Thu, 18 Nov 2021 23:30:26 +0000

The Indiana Secretary of State business search is a great search that has many different search options. You can search for any kind of registered business entity, such as LLC, corporations, or partnerships.

Here is the official search page.

While it is not required to be registered with the IN Secretary of State to do smaller business before you do anything like open a business checking account or get a business loan, you must register with the state. Because it is public you can find business owner information for any business registered in the state of Indiana.

I rate it 7 out of 10

Pros:

Search by officer and registered agent
Search by other neat things like entity status and entity type
View officers and documents from details page
Search UI is pretty

Cons

reCaptcha on every search is a pain
Some searches can take a LONG time

Indiana Secretary of State Entity Search

Look how pretty the Indiana Secretary of State business search is! There are quite a few search options, including search by the registered agent and officer/incorporator/governing person.

Let’s give it a try. The captcha is always kind of a pain but every time I checked this one I didn’t have to select any images. The load time took 2.21 seconds. Here are the results:

There is quite a bit of information here. The most important things are probably the business name, type, and status. It’s nice to also have the office address, I guess. Maybe if you wanted to find one close to you?

Here’s what it looks like when I enter the details page. It has more business detail information, including filing date, and then great information such as officer and registered agent data.

It also has a section here for business documents such as Articles of Organization and what looks like an annual Business Entity Report.

I did open up an Articles of Organization and it didn’t look like it had any cool information like phone numbers or email addresses.

Search by Incorporator

Now to try out the search by incorporator. I searched by “Frank” and it took a lot longer to load at 1.2 minutes!

And…the results look the same. 10k+ results, though! No wonder it took so long to load. The interesting thing is not all of these results are loaded right now. If I hit the next page button it still takes forever to load page two. Ouch.

And that’s about it for Indiana.

Extra Credit

For all of my power users out there. You can access any of this business data via API. This can be handy for things like:

Business verification
KYC/KYB (know your business) anti-money laundering
Business credit check
Finding business owner information

It’s as simple as this:

let url = `https://apigateway.cobaltintelligence.com/v1/search?searchQuery=${encodeURIComponent(businessName)}&state=${state}`;

const axiosResponse = await axios.get(url, {
    headers: {
        'x-api-key': this.apiKey
    }
});

return axiosResponse.data;

Get a free Secretary of State API key here.

Here are some technical posts on how to get information from the Indiana Secretary of State:

The post Indiana Secretary of State Business Search appeared first on Cobalt Intelligence.

Oklahoma Secretary of State Business Search

Jordan Hansen — Thu, 18 Nov 2021 14:14:23 +0000

Welcome to the page that is going to talk all about the Oklahoma Secretary of State business search. First point. Should Secretary of State be capitalized? I do not know.

The Oklahoma Secretary of State entity search will allow you to search for LLCs, corporations, partnerships, or any other entity that is registered with the state of Oklahoma. Here is the official search page.

Oklahoma Secretary of State Entity Search

Using the Oklahoma Secretary of State business search The advanced search may not look as fancy as other states but it does allow to do quite a bit, including search by active status, check business name availability, and search by individual name.

Making my search for “pizza” with active businesses only took 2.32 seconds. Not too bad. Here’s the list.

Classic information here. Entity type, status, and registered agent. I find it interesting here that it has a “Name Type” field. I’m not sure what that means. I haven’t ever seen it return anything other than “Tradename”.

Clicking the filing number takes us into the details page. It is one of the more sparse details page.

This business is supposedly active but you can see above it has an “OTC Suspension”. I believe this means that the business is currently active and in good status with the Oklahoma Secretary of State but is not in good shape with the Oklahoma Tax Commission.

Order Documents?

I clicked in to order documents to see what it would take. It asked for my name and email address and then the documents I wanted. Looks like they charge per document. Ouch.

Does something with a charge quality as public under the Freedom of Information Act? I don’t know. I’m sure it started as an administration fee but with this kind of thing being easily fully automated, I wonder how applicable it is.

Search by Person Name Weirdness

Searching by person name is an interesting one in Oklahoma. You can do it but the person’s name isn’t visible on the details page. So you end up with a situation like this:

The owner/officer of this company is “Rosa Anguiano Garcia” but that name doesn’t show up anywhere on the details page. You can see the registered agent name but it is definitely not Rosa.

Extra Credit

So you can find out businesses owned by an individual but if you are looking at that same business later, you will not know that they own that business. Registered agent is sometimes the primary officer/owner but it certainly does not have to be.

For all of my power users out there. You can access any of this business data via API. This can be handy for things like:

Business verification
KYC/KYB (know your business) anti-money laundering
Business credit check
Finding business owner information

It’s as simple as this:

let url = `https://apigateway.cobaltintelligence.com/v1/search?searchQuery=${encodeURIComponent(businessName)}&state=${state}`;

const axiosResponse = await axios.get(url, {
    headers: {
        'x-api-key': this.apiKey
    }
});

return axiosResponse.data;

Get a free Secretary of State API key here.

Want know how to do some web scraping of the Oklahoma Secretary of State, here’s a helpful video.

And if you are looking to learn how to get newly registered businesses with the state of Oklahoma, here it is:

The post Oklahoma Secretary of State Business Search appeared first on Cobalt Intelligence.

Mississippi Secretary of State Business Search

Jordan Hansen — Thu, 18 Nov 2021 12:38:19 +0000

The Mississippi Secretary of State business search is a great search that has many different search options. You can search for any kind of registered business entity, such as LLC, corporations, or partnerships.

Here is the official search page.

While it is not required to be registered with the MS Secretary of State to do smaller business before you do anything like open a business checking account or get a business loan, you must register with the state. Because it is public you can find business owner information for any business registered in the state of Mississippi.

Mississippi Secretary of State Entity Search

Let’s start searching! I started off searching for pizza. The results came back fast, in maybe 1.5 seconds. Check out the list.

You can see that from their results page it lists a lot of helpful information, notably the entity type, status, and its initial filing date. I especially like that last one, “Pizza Doctor, LLC”. Let’s check that one out.

Clicking the “Details” button opens a modal with additional business information.

This has it all. Business address as well as the address of the officers and the registered agent.

For clarification, a registered agent is someone who is responsible for receiving legal documents. This is often the owner but can also be a third party service, accountant, or lawyer. Officers and directors would be the main decisions makers for the business.

Other Cool Stuff

Mississippi also allows you to view all documents that have been filed with the state. This has things like the articles of organization and annual reports.

Now, let’s real talk. If you open up the filed documents and look for a formation form, you’ll see something interesting in it. A nice, plump business email address for the ower. I blurred it out here. I don’t know why, it’s public information. If you want it, go get it!

One other noteworthy thing is the ability to opt-in to email updates. This is a good way to keep track of when a business has gone delinquent or has become active. Thanks, Mississippi!

Find Business Owner Information

Mississippi offers some different search options compared to other Secretary of States. Some states have the option to search by date range and entity status (looking at you, Washington).

Mississippi has the cool ability to search by both Officer Name and Registered agent. This is a handy feature if you’re looking all of the businesses that an individual might own or be a part of.

Extra Credit

For all of my power users out there. You can access any of this business data via API. This can be handy for things like:

Business verification
KYC/KYB (know your business) anti-money laundering
Business credit check
Finding business owner information

It’s as simple as this:

let url = `https://apigateway.cobaltintelligence.com/v1/search?searchQuery=${encodeURIComponent(businessName)}&state=${state}`;

const axiosResponse = await axios.get(url, {
    headers: {
        'x-api-key': this.apiKey
    }
});

return axiosResponse.data;

Get a free Secretary of State API key here.

The post Mississippi Secretary of State Business Search appeared first on Cobalt Intelligence.

Getting great New Hampshire business leads

Jordan Hansen — Thu, 14 Oct 2021 20:00:53 +0000

Demo code

New Hampshire, today we come to vanquish you (in terms of getting business leads from you).

New Hampshire Secretary of State

Not a particularly robust search, New Hampshire’s business search allows you to search by business name with a “Contains” filter. It also has a captcha, which makes us to have to be a bit more creative with our web scraping.

Side note. There are tools that you can use to solve captchas in an automated fashion. I use 2captcha and am really happy with it. In fact, I’ve solved 85,516 captchas with them over the last 6 months.

Check them out with my shiny affiliate link here – https://2captcha.com?from=7390140

While we’re going to do the first step of this process: seeing how many pages of results we’re going to get.

So we type in the industry/keyword we are looking to target, select contains, and enter the captcha (remember, it is case sensitive). Hitting search after will give us our results after a short load time.

There. Searching for pizza returned 2,150 results and, more importantly for our purpose here, 86 pages. Make note of that!

Code setup

This is the part where it’s going to take a little bit of code. You will need to download git and Nodejs in order to make this work and not be too intimidated by code. It also would be helpful to have some kind of code editor, such as VS Code.

Once you have those go sign up for a free API key here. Right now Cobalt Intelligence gives 100 free requests per month and after that it’s very inexpensive for additional requests.

Now, clone the code from the github respository, install all dependencies with npm i, and open the code in VS code.

Once you’ve signed in you’ll have an API key. Copy it from here:

With that API key, we’ll want to open .sample.env and replace the dummy API key in there with your actual API key.

Run that code!

Now our code should be all set. We need to make sure to go and update the number of iterations based on the number of pages that we remembered from above.

Update this to the number of pages

With that updated we make sure that await getAllBusinesses(fileName); is commented in. It should look like this:

Run that baby with npm start. This will start Puppeteer and open a browser. You have 30 seconds to enter the keyword you are looking for, select “Contains”, do the captcha, and click on search. After that, the code will take over and do the rest. It’ll take a bit of time depending on how many pages your keyword has. So, go get some snickers or take a nap.

Step 2 of code: Getting the business leads

Okay, so the above part of the script will give us a list businesses in New Hampshire that are “Active” or in “Good Standing”. Now we need to run it through the Secretary of State API to get the owner information and contact information.

All you need to do is comment out the await getAllBusinesses(fileName); line and comment in the await getBusinessDetails(fileName); line. It should look like this:

This will run the rest of the code against the Secretary of State API. This could also take some time so let it run. When you get back, you should see a CSV that has the leads you’re looking for. Use it and love it.

Demo code

And here we have the video transcript:

Hello there. My name is Jordan Hansen. I’m from cobalt intelligence. And today I’m going to talk a little bit more about getting business data from New Hampshire. It’s gonna be a little bit different today. It’s going to be kind of a trick where we’re going to try to see, oh, how many businesses can. From New Hampshire from a specific category or keyword.

And the cool thing is New Hampshire. We can get through the secretary of state API here that cobalt intelligence provides. And I’m with Coldwell intelligence, you can get email addresses and phone numbers, so you can find the owner of the business and then be able to reach out to them with good contact information.

So here’s the New Hampshire secretary of state they have. It’s pretty simple. It’s not as cool as robust as some of the other ones we’ve seen to start off with. We want to go over here and. Total. We want to pick a category so we could say something like, um, let’s say we want rush or pizza pizza places or, Hmm, what’s another good one.

Um, insurance, let’s say insurance agents scare. We go over here like this, we say SDN about this capture hat and case sensitive case in CV matters here. And you hit search. Now it’s going to come over here. This is the first step you have to do. You have to come over and search for what you want, and you have to look at how many pages there are.

So for this one, there’s 238 pages. So you scroll down here to this part right here, need to update the end of the loop to match the number of pages. What do you say? We make that font even bigger. There we go. Pages. So this one would go the two them. Like that now that’s too many for this demo. Cause that will take hours, maybe hours.

I dunno a while it take a while. So what we’re going to do is we’re going to just search for something weird. We’ll just say sword, let’s say eight H N V K. We’ll search for it.

There we go. And our total of two, two pages. Okay. That makes easier. And now, now we’re gonna let the thing do its work. So this code will be, there’s a link to this code in the comments. You can go ahead and clone it and use it. Come over here. Once you update that page, the first thing is to do is update the length of the pages right there, and then you run it and it’s going to pop up.

I have this browser, it’s going to use puppet here. It’s going to pop up something we’re going to have to type in some stuff like the capture and what we’re searching for. Can we come over here work and we’ve got a sword having a capture eight, you had 30 seconds to do this G to D J search. And then once it gets it, it’ll go ahead and start paginating until the end.

So right there, bam. So you just got the first businesses. It’s going to the next one. Oh, sorry. It’s going to start in the first one right here. I give it 10 seconds for the page to load. Cause I can’t really tell what each one is. So it’s going to be 10 seconds between each page. And they’ve seen now it’s on the second one and you can see over here, see, it’s collecting the database IDs, all these businesses bam done.

So we found all these businesses right here. It actually only found eight. Now that may be weird, like, oh, why are there only eight? Well, I’m filtering out by ones that are active or in good state. If you didn’t want that, you could come over here and we move this if statement, um, and then you wouldn’t have a problem, right?

So you come over here and say, okay, active, good standing, whatever. So now that we have those, now we use the secretary of state EPA. Now you have to get an API K for the, you have to get an API key first, come over here to cobalt intelligence, get a free API key. It costs per request, but you get three requests per month.

So you can come over and use those if you want. Um, I didn’t want to print like this. You go over to the sample that in file right here. No update your API key in here and you’ll rename this file to dot M right there. So we put the APA API key in there. Then you just run this thing right here. And what that’ll do is that an open up the file that we just created oh, right down here to open the file and then it’s going to loop through them all.

And it’s going to get the data for those ones, these specific ones. So everything you can find for these active. So, and it’s going to filter them out based on the ones that have email addresses. So if there’s no email address or if it says none, that’s not going to push it into the CSV. So now we go over here, we run this guy.

Yep. Wait, give business details. Here we go. Ready? It’s going. There’s eight total. That’s correct. Now it should start looping. Bam. We got data, data, data, right? Looking good. Looking. Okay. So that means four businesses because four of them have email addresses. You can look over here, this one, no email address.

So we didn’t push it in there. We’re not interested. Maybe you are. And then you could change that if condition, no email address on that one, but there is one here. It looks you right there. And there was one here and there is one. So that’s there we go. So we’re getting email addresses now we’re done. So we’ll go here full business information.

You can see them all here with the email addresses. That’s great. And now you can just, uh, you could market to them however you want it or reach out and do whatever you wanted. And this is a way to get targeted business information in the area of New Hampshire. So you just need, um, this code and the cobalt intelligence API key.

And there you go. Go from there. That’s it. Thank you.

The post Getting great New Hampshire business leads appeared first on Cobalt Intelligence.

Virginia business leads for pennies

Jordan Hansen — Mon, 11 Oct 2021 06:31:00 +0000

Demo code

The goal is to get some great, targeted Virginia business leads for pennies.

Here we gooooooooooooooo!

Virginia Secretary of State Search

The Virginia Secretary of State search is incredibly robust. Look at all of the awesome search options you have.

The options I tend to use are searching by “Contains”, changing the “Entity Status” to “Active”, and possibly changing the “Date of Formation” range.

Searching a large amount will bring up a prompt that asks if you want to few all results. You definitely want to say “yes” to this! It will take longer to load but you want all of the results.

Once you see the results, you’ll want to go down to where the magic happens, the export to CSV button.

Click it and it will start working on your download. The bigger the list the longer it takes to process and download. But I have downloaded 10s of thousands of businesses without any problem.

To the code!

Once you have those go sign up for a free API key here. Right now Cobalt Intelligence gives 100 free requests per month and after that it’s very inexpensive for additional requests.

Now, clone the code from the github respository, install all dependencies with npm i, and open the code in VS code.

Once you’ve signed in you’ll have an API key. Copy it from here:

With that API key, we’ll want to open .sample.env and replace the dummy API key in there with your actual API key.

Now copy the file you downloaded from the Virginia Secretary of State into the base folder of the repository we cloned above. Change the file name in src/index.ts in this line (it’ll be towards the top) const businesses = await csvtojson().fromFile('pizza businesses.csv'); to be whatever your file name is that you downloaded (probably something like “Search Results.csv”).

If you want to get fancy, you can also change the ouput file name towards the bottom where it says fs.writeFileSync('Full pizza businesses found.csv', csv); to whatever you want your csv file to be named.

Now…just run npm start. It’ll take some time to process the whole file depending on how many results you have. I wouldn’t be surprised if it was 7-10 seconds per business in that list. It will log out it’s progress as it goes with each business it finds.

Conclusion

Once it’s complete you’ll have a newly created file full of emails and phone numbers.

What now?

Well, reach out to them! Make their lives better with your awesome product!

Video Transcript:

All right. Good morning. Um, hello, how you doing? I’m Jordan Hanson. I’m from cobalt intelligence. Today. We are going to show some cool tricks of how to get some cool business data from the Virginia secretary of state. And if you combine it with the cobalt intelligence secretary of state API, you can get cool contact information.

Um, now again, this is where it is. You need API key I’m over here. You will need an API key to be able to do this today. Um, you can get a free API. It comes with a hundred requests per month. Um, so let’s go over here to Virginia and show how this works. Okay. So Virginia is a cool state that allows you to do cool things like their search is really good.

Um, you can search by all bunch of different things. And today I’m going to search by, let’s say, I’m going to search by contains. I’m going to search by, I would say, really have a product that’s really good for, uh, video games. Let’s see how many businesses have video. You’re gaming, maybe in the title, let’s see, it’s a C and I want them to be active.

I want these businesses that are active right here. So you can search, you can search word. I want, which by industry now they’re industry, most people it’s, whatever the individual selects. So a lot of times they just put general. Um, but they have other ones too. There’s insurance, other things like that. So that’s what that video gave me.

We’re going to look for one that has like 10, we’re looking for a small one just for it to show the sample. But like, I’ve done this with thousands before. You can do it without any problems. So it takes a little bit of time for, to search. But what we’re going to do, and I’m going to have the code. I have the code over here, what we’re going to do.

And, um, we’re going to, uh, this could be public. We’ll have the repository out there so you can really just clone it, put your API key in it and run it. Okay. Well, we have one. I wanted more than that. That’s go a little more. Let’s get more than that. What we’ll do is we’ll go like. Pizza and we’ll do it. We want him to have the word pizza and it’s going to be active and we’re going to formation data of in the last, um, since August 1st of this year.

So two months and see how big we get. We do have a fair amount. 26. Okay. 26. Now look at the bottom here. This is the cool part. And then here, you can see, you can see the export buttons right here. Now we want to export to a CSV. So this is cool. This will give us a list of all the businesses in a CSV, and you can download all of them.

Like it says it’s limited to 5,000 records export, obviously for PBS. So I’ve exported thousands. You can have 11,000 records, whatever. Any of that that’s put over here, we’re going to go like this. There’s our results. See. All the cool pizza companies right here. And then what we’re going to do

is we’re gonna come over here and we’re going to rename it to, um, pizza, pizza businesses

right there, and then we’re gonna call it what’d I call that yeah. Pizza businesses, all lower case. Uh, businesses. Okay. So then I got my list. So we’re going to come over here. We’re going to use this package called CSP. The J shot. Jason wants me to take the CSV. It’s going to convert it into an array of Jason objects, and then we’re going to come down.

It’s going to loop through the businesses. Uh, first we’re initialized the API, which we’ve, um, there’s going to be, I have loads of guides of how to use the API, but we’re going to import it from the cobalt and SDK. I’m a pass in my API key. I started here then loop through these businesses and each one I call the API key and I get to get details by SOS ID.

Now, inside that thing, you can see, we have a entity ID right here, uh, and to ID, right. So we’re looping through we’re passing business entity ID and a state, which is Virginia. Now the cool thing about Virginia, the secretary of state, we can get emails and phone numbers for these items, these places that are really good.

So let’s run it here and see what we get. So we have 26. It might take a little bit, um, we’ll run it right now. We’ll say, Hey for pizza businesses bound and we’ll go MPM start. Oh, we probably don’t have it. Hold on. Let me read my script here. Yeah, this is a brand new start and we’ll write about TSC and present ampersand node, this index just like that.

And then we’re going to run it. Now. This will start parsing through this data. Okay. Just kidding. All the time I get it spelled right. I literally spelled it with. How does pizza businesses? I should’ve just left it and we named it cause that style right there, busy misses. All right. So at 26, it’s going to go through a loop through them.

That’s a whole, we have, now there is some way for advanced people. You can go ahead and do some kind of asynchronous where you can fire up multiple ones, but band looks, you were already going by phone numbers, email addresses right here. There. Email justice. So we can call, you can contact these people and reach out.

Let’s say we have a really good product for them. Let’s say we want to order pizza from them. We want to check out all these different things with pizza, whatever. Right. So we’ll loop through each one I haven’t set up. So if it has an email and the email is not this N a um, then I’m going to go ahead and push it into my full businesses.

And then after that, I’m going to put it into this array, but I’m going to convert it to CSV and then put it right into a file. That’s it pretty simple, the very simple way to get in there, get some cool business data for me. What am I doing? Cool business data for Virginia, any industry you’re looking for, insurance, software financing, whatever, what accounting, any of those things, you can get a huge list of them.

Get email addresses. This is a cool way to do it. So that’s it. This will finish up. It’ll create the CSV. I kinda it’s gotta be close. How many things we’ve done.

Oh yeah. We’re close to the end. I think we have multiple of these. Oh, maybe let’s see. Sergeant Pepper’s pizza. Oh yeah. We’re close here. 16. So of those 26, we had 16 with good email addresses and now we could talk to them and we doubt and say, Hey, we’ve got this awesome thing we want to talk to you about.

And we go from there and that, and then we create Lexi right here. Bold pizza businesses. All creative with phone numbers, email addresses. We’re good to go. So that’s it. And this again, this is Jordan Hanson, cobalt intelligence. If you need an API key, go here to cobalt intelligence, get one and, um, uses data.

Hopefully you can find some really cool stuff. Thanks.

The post Virginia business leads for pennies appeared first on Cobalt Intelligence.

Web scraping the New Hampshire Secretary of State for business leads

Jordan Hansen — Tue, 07 Sep 2021 12:47:00 +0000

Full video walkthrough

This video goes over using Puppeteer to scrape business data from the New Hampshire Secretary of State. New Hampshire has a normal image captcha and we solve this by using 2Captcha.

This adds getting business data from Nebraska to our Secretary of State API. If you’d like an API key, get one here – https://cobaltintelligence.com/secretary-of-state

I’ve used 2Captcha for a few years now and it’s been a great, very affordable service for solving captchas. 2Captcha affiliate link – https://2captcha.com?from=7390140

The post Web scraping the New Hampshire Secretary of State for business data appeared first on Cobalt Intelligence.

API Key on AWS API Gateway

Jordan Hansen — Mon, 01 Mar 2021 13:14:00 +0000

This walks through how to set up an API key on AWS API Gateway. API keys on AWS API Gateway go in tandem with Usage plans. Because of that, I also walk through setting up a Usage plan.

This all combos with an AWS Lambda function that is called from the AWS API Gateway.

I am using this on AWS in order to create an API that allows you get business details from Secretary of States around the country.

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome web data. Learn more at Cobalt Intelligence!

The post API Key on AWS API Gateway appeared first on Javascript Web Scraping Guy.

AWS API Gateway and Lambda

Jordan Hansen — Mon, 22 Feb 2021 13:21:00 +0000

Most of this process is documented really well by AWS here – https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html

The most important parts that I believe are missed are when it comes to using the custom domain name. You have to set up a CNAME record to the API Gateway domain name shown in the Custom domain names section.

You do NOT want to point your CNAME to the test URL provided. Also…the stage is only used in the test URL, not the actual one used in your custom domain name API mapping.

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome web data. Learn more at Cobalt Intelligence!

The post AWS API Gateway and Lambda appeared first on Javascript Web Scraping Guy.

Puppeteer on Lambda! Updated and better.

Jordan Hansen — Mon, 15 Feb 2021 13:09:00 +0000

Demo code here

Uploading directly to Lambda has a size limit of 50MB.

This makes it difficult for Puppeteer, which is a very large library. Uploading your code to s3 and then updating your function from there increases this size limit to (I think) 250MB!

I use 7zip and typescript and here is my scripts that I use to upload to Lambda.

"scripts": {
    "zip": "tsc && 7z a -r function.zip ./dist/* node_modules/",
    "sendToLambda": "npm run zip && aws s3 cp function.zip s3://chrome-aws && aws lambda update-function-code --function-name puppeteer-on-lambda --s3-bucket chrome-aws --s3-key function.zip"
  },

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome web data. Learn more at Cobalt Intelligence!

The post Puppeteer on Lambda! Updated and better. appeared first on Javascript Web Scraping Guy.

Scraping Utah, Part 2

Jordan Hansen — Mon, 08 Feb 2021 23:11:00 +0000

The struggle with Utah secretary of state continues.

And it seriously is a struggle. I’m having real concerns about the successful completion of this scrape.

2captcha continues to be a star, though. Here’s my affiliate link if you want a great captcha solver – https://2captcha.com?from=7390140

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome web data. Learn more at Cobalt Intelligence!

The post Scraping Utah, Part 2 appeared first on Javascript Web Scraping Guy.

How to Handle Puppeteer’s waitFor Deprecation

Jordan Hansen — Mon, 01 Feb 2021 23:05:00 +0000

Puppeteer has deprecated waitFor! But it’s not as terrible as you think.

There it is! Looking good!

Looking for business leads?

Using the techniques talked about here at javascriptwebscrapingguy.com, we’ve been able to launch a way to access awesome web data. Learn more at Cobalt Intelligence!

The post How to Handle Puppeteer’s waitFor Deprecation appeared first on Javascript Web Scraping Guy.