DEV Community: Joseph Lim

The VC's Guide to Uncovering Stealth Startups

Joseph Lim — Wed, 21 Jun 2023 08:51:26 +0000

How can you identify a likely founder so that you can be the earliest to invest in their success?

In this article, we'll answer this question and show you several specific queries you can plug into the Proxycurl Person Search Endpoint to search for potential founders. Our intent is to demonstrate how you can add sourcing to your stack, rather than needing to have analysts do this manually.

General method

We will make the assumption that founders tend to have strong demographic trends in certain categories, such as:

Attended certain universities.
Worked at certain companies in the past.
Held certain past job titles (such as were previously founders).

You can leverage these trends along with a powerful Person Search API like Proxycurl's to make bulk search queries that return rich lists of potential founders whose careers you can then monitor. If such a person moves to start a new company - possibly a stealth startup - you'll be aware of it.

Searching for employees of "Stealth Startup"

A somewhat naive approach to the problem is to search for employees of "Stealth Startup." Fortunately, the naive approach is valid because LinkedIn (both helpfully and somewhat amusingly) recognizes "Stealth Startup" as a legitimate company. It has job listings and everything.

There are a couple of specific variations on this name that have a lot of employees:

And several others. We're going to run a Person Search Endpoint query with a regex on these, and we want to pick up a lot of results, but we don't want to grab too many false positives; for example Stealth Management Group LLC is not a stealth startup.

For our regex, let's look for either an exact match of Stealth (^Stealth$) or require that both Stealth and Startup be in the name, possibly with some string in between them. We can accomplish that with the following complete regex:

    'current_company_name': '(^Stealth$)|(.*Stealth.*Startup.*)',

We'll leave case sensitivity on.

It turns out that there are too many LinkedIn employees of stealth startups for this to be a restrictive enough condition to be searched on its own. That's exciting because it means we're working with a large sample size. So let's move on to the next part before we run any code.

Segment potential founders based on their background

We can accomplish this task in a few ways, and depending on how selective you want to be in your search, you can use these methods all at once or one at a time. Additionally, you can be more or less selective within each method; for example, when searching for graduates of particular universities, you may decide to look at only a single Ivy League school, whereas here, we'll include them all.

We'll include one constant in every search here: Duration of tenure at the stealth startup. We're primarily interested in the first couple years of a stealth startup's existence, so we'll set current_role_after to 2021-04-01.

Filter by education background

The first method we'll try is filtering for potential founders by their education background. You may have a school or set of schools in mind, but we'll take the set of Ivy League universities as an example. The regex for this is pretty straightforward, if a bit long:

    'education_school_name': '^(Brown University|Columbia University|Cornell University|Dartmouth College|Harvard University|University of Pennsylvania|Princeton University|Yale University)$',

Again, we want case sensitivity here, so we'll omit the optional (?i) flag. And now, with two parameters specified, we are restricting our query enough that we can run some code! Here is an example working Python script. (As usual, with queries that return people, we won't print any results, but you can run it yourself if you like.)

import json, os, requests
api_key = os.environ['PROXYCURL_API_KEY']
headers = {'Authorization': 'Bearer ' + api_key}

api_endpoint = 'https://nubela.co/proxycurl/api/search/person'
params = {
    'current_company_name': '(^Stealth$)|(.*Stealth.*Startup.*)',
    'current_role_after': '2021-04-01',
    'education_school_name': '^(Brown University|Columbia University|Cornell University|Dartmouth College|Harvard University|University of Pennsylvania|Princeton University|Yale University)$',
}
response = requests.get(api_endpoint, params=params, headers=headers)
print(json.dumps(response.json()))

Filter by past experience

After their education, people go to work, and you can use people's work histories to identify interesting people to keep track of. Historically, the acronym in tech has been "FAANG," and you might have your own particular set of companies you're interested in (for example, perhaps you want to include Microsoft). Still, we'll stick with the classic version for this demo.

Now, there are two ways to approach this given that we have a list of companies to OR together:

Use the past_company_linkedin_profile_url parameter.
- Pros: This field guarantees an exact match.
- Cons: It does not accept a regex value, so we can only search for one company at a time. Therefore, we have to perform multiple searches and combine the results.
Use the past_company_name field.
- Pros: This field accepts a regex value, so we can search for multiple companies at a time, using ^ and $ with case sensitivity to ensure the name exact matches.
- Cons: While we can ensure that the name exact matches, we might pick up some false positives; for example, there are a couple other companies named Apple besides the FAANG Apple that we actually mean.

Which method should we choose? It depends on whether there's a reason not to use method 1. Here's why you might want to avoid method 1:

Multiple cases of needing to run multiple queries due to ORs with different fields is a reason to avoid method 1.
Having lots of parameters in your query, so that it's more likely that a case of the "wrong Apple" showing up is more likely user input error than a wrong data point is a reason to avoid method 1.
Running a very small sample size, so that the cost to run any query at all is high, is a reason to avoid method 1.

Given this analysis, I think option 2 is preferable for this particular use case due to the second bullet point. However, this is not a be-all-end-all decision. Sometimes, you will certainly prefer option 1, and you should always consider which option is better for any decision point. When possible, make a query that returns a hundred or so results and step through them manually before committing.

With that decided, we can run the following code:

import json, os, requests
api_key = os.environ['PROXYCURL_API_KEY']
headers = {'Authorization': 'Bearer ' + api_key}

api_endpoint = 'https://nubela.co/proxycurl/api/search/person'
params = {
    'current_company_name': '(^Stealth$)|(.*Stealth.*Startup.*)',
    'current_role_after': '2021-04-01',
    'past_company_name': '^(Facebook|Apple|Amazon|Netflix|Google)$'
}
response = requests.get(api_endpoint, params=params, headers=headers)
print(json.dumps(response.json()))

Notice we're now discarding the segment from the previous section, but if we wanted to we could combine them and specify both an education_school_name and a past_company_name at the same time.

Filter by past role title

Finally, let's look for "serial entrepreneurs." We haven't been requiring that people list themselves as founders of their current stealth startup, just that they work at one. The idea is that anyone currently working at a stealth startup is someone who you might want to keep an eye on - either for their current company or their next one. But we can select for people likely to go on to be a founder based on whether they've been one in the past.

Here's the query, this time using the (?i) flag in case someone was a "Co-founder":

import json, os, requests
api_key = os.environ['PROXYCURL_API_KEY']
headers = {'Authorization': 'Bearer ' + api_key}

api_endpoint = 'https://nubela.co/proxycurl/api/search/person'
params = {
    'current_company_name': '(^Stealth$)|(.*Stealth.*Startup.*)',
    'current_role_after': '2021-04-01',
    'past_role_title': '(?i)founder'
}
response = requests.get(api_endpoint, params=params, headers=headers)
print(json.dumps(response.json()))

Finding founders who don't list "Stealth Startup"

That was all great if the founder actually listed their current company as a recognizable regex matching .*Stealth.*Startup.* somehow, but what if they're calling it something else entirely? For example, they could have written "Working on something cool" or "Can't wait to tell you guys."

In the last section, we've profiled a few segments of likely founders (at least, this is the hypothesis). We can combine these - and, critically, remove the ones we've already looked at by excluding the "Stealth Startup" regex we already looked at - to find candidates likely to be working at "interesting" companies at some point in their careers. The goal here is to curate a short list of people who are:

Possibly working at an interesting company right now; or
Potential founders of the next wave of startups.

There is one limitation here. Proxycurl's Search API was released pretty recently as a version 1.0 product. Therefore, one field we might want to add to our search doesn't exist yet: current_company_employee_count_max. If you're reading this post a while after its publication date, you may be able to disregard this caveat - check our docs for the Person Search Endpoint to see if this parameter exists. It's on our to-do list. But right now it's unavailable, and we'll have to make do without it.

If we combine all the restrictions we looked at before, we only get nine results total. So we'll have to relax the conditions a bit. We'll only look at either graduates of an Ivy League school or alumni of a FAANG company (rather than both simultaneously).

Because the code snippets are very similar, and the condition for the second was provided above, I'll only demonstrate the first here, and the second is left as an exercise to the reader.

import json, os, requests
api_key = os.environ['PROXYCURL_API_KEY']
headers = {'Authorization': 'Bearer ' + api_key}

api_endpoint = 'https://nubela.co/proxycurl/api/search/person'
params = {
    'current_company_name': '^(?!(^Stealth$)|(.*Stealth.*Startup.*))',
    'current_role_after': '2021-04-01',
    'education_school_name': '^(Brown University|Columbia University|Cornell University|Dartmouth College|Harvard University|University of Pennsylvania|Princeton University|Yale University)$',
    'past_role_title': '(?i)founder'
}
response = requests.get(api_endpoint, params=params, headers=headers)
print(json.dumps(response.json()))

What next?

Now you have some candidates. That's a great first step, but if you source candidates and do nothing with this data, you still have nothing. So you will need to continue to monitor your list of candidates and eventually reach out to them.

Monitoring

You can monitor candidates using the Person Profile Endpoint. If their company name changes from Stealth Startup to MyCompany BioLabs then you can reach out. Be sure to include the parameter use_cache=if-recent so that you're always getting the most recent data, and you can get updated data once a month. Here's how a sample query might look with just a little bit of data processing:

import os, requests

api_key = os.environ['PROXYCURL_API_KEY']
headers = {'Authorization': 'Bearer ' + api_key}

api_endpoint = 'https://nubela.co/proxycurl/api/v2/linkedin'
params = {
    'url': 'https://www.linkedin.com/in/johnrmarty/',
    'use_cache': 'if-recent',
}
response = requests.get(api_endpoint, params=params, headers=headers)
result = response.json()
current_companies = [exp['company'] for exp in result['experiences'] if exp['ends_at'] is None]
print(current_companies)

And the result:

['Freedom Fund Real Estate', 'Mindset Reset Podcast', 'Project 1B', 'YouTube', 'YouTube']

Here's what we did:

Query the Person Profile Endpoint.
Extract the JSON result.
Zero in on the "experiences" section.
Select only the current experiences.
Select the company name for each of those current experiences.
Print out a list of them.

If you were doing this as part of a pipeline, you'd have past data to compare to, and you'd probably want to keep track of some additional factors as well like date joined and maybe description, to check if this is likely a name change (e.g. from "Stealth Startup") or a brand-new venture. You'd also want to write this to a file or database rather than printing it.

But as far as a demo goes, not so bad!

Contacting

The final step in your pipeline is going to be contacting the prospect. We have another endpoint dedicated to this purpose, the Personal Email Lookup Endpoint. Here's how you might use it to finally reach out to a lead you've been following:

import json, os, requests
api_key = os.environ['PROXYCURL_API_KEY']
headers = {'Authorization': 'Bearer ' + api_key}

api_endpoint = 'https://nubela.co/proxycurl/api/contact-api/personal-email'
params = {
    'linkedin_profile_url': 'https://sg.linkedin.com/in/williamhgates',
    'email_validation': 'include'
}
response = requests.get(api_endpoint, params=params, headers=headers)
print(json.dumps(response.json()))

Note that for the email_validation parameter we used include. This parameter requires the endpoint to perform email validation. If exclude is used instead (or the parameter is omitted), there will be no email validation.

The endpoint will return personal email addresses of your prospects, and your output will look like this:

{
    "emails": [
        "personal.email1@gmail.com",
        "personal.email2@yahoo.com"
    ],
    "invalid_emails": [
        "personal.email3.doesnotexist@gmail.com"
    ]
}

In case you're wondering why we aren't using the Work Email Lookup Endpoint the answer is: Usually we would! But in this case we're looking at brand-new companies and/or stealth startups. These companies are unlikely to have domains added, much less work emails available. You're better off contacting via personal email in this particular situation.

TL;DR

This article is long enough that it deserves a TL;DR section.

If you're a VC or investment company, you may currently have many manual steps in your pipeline for identifying potential founders and contacting them.

What if you could fully automate this pipeline? With Proxycurl APIs, you can! The Person Search Endpoint is the core of the method. You will use this endpoint to search for current employees of stealth startups, filtered based on various criteria - we suggest some with code samples, but you may have your own that you'd prefer to use.

After that, the rest of the pipeline can be accomplished using the Person Profile Endpoint and the Personal Email Lookup Endpoint. Code samples for these are provided as well.

Conclusion

How else might we use Search or even the other Proxycurl API endpoints to find interesting founders of startups? We have a few tricks left up our sleeves - but maybe we've also inspired you along the way! If you want to be featured in one of our blog posts, reach out at hello@nubela.co. Or if you'd rather keep your discovery to yourself, register an account and make a credit top-up so you can get started right away getting a leg up over the competition.

1 in 3 Ex-Google Employees In $15 Billion Worth Of Startups [2023 Layoffs]

Joseph Lim — Fri, 09 Jun 2023 08:11:03 +0000

From a life of corporate toiling to startup stardom, droves of ex-Googlers have turned their talents to the startup world. The result? A collective $15 Billion worth of venture-funded startups now call ex-Googlers their own.

Who are these ex-Googlers exactly? Which geographical locations are they from? What kind of industries did the other 60+% go to? Join us as we uncover the secrets of life-beyond-layoff for thousands of these former Google employees, and see if we can unearth more insights using cold, hard data.

At Proxycurl, we are in the data enrichment tech business where we enrich people and company profiles with structured data including people data, company/school data, contact & jobs data. Clients that use our APIs & databases include VCs, banks, startups, sales organizations, HRs & more.

We hope you will find our insights useful. But that’s not all - as a special treat, we're offering a never-seen-before giveaway of the entire dataset of ex-Googlers scraped using our products. And this is just the beginning - we have a series of content (and giveaways) coming up that cover the other affected big tech companies, including Meta, Netflix, and Amazon. So be sure to follow us for more insider insights, expert analysis, and the latest news from the tech industry.

Content

Methodology
Part I: People Profiles Of 2,762 Ex-Google Employees - Who Are They?
- Every 2 Laid-Off Googlers, >1 Is From The States (56%)
- Software Engineers Were Most Affected: 30% Of All Laid-Off Employees
Part II: 747 Ex-Googlers Found New Jobs - At Which Companies?
- 747 Ex-Googlers Landed New Jobs / Founded New Companies
- 58% Of New Companies Are In Technology & IT, Includes Software Development
Part III: Billions Worth Of Impact Made By Just One Third Of Ex-Googlers
- One-Third (or 205) Ex-Googlers Joined Startups
- Equal Proportion Of Startups In The Same Industries As All-Company Universe - 55% In Technology & IT
- Close To 70% Of Ventures Are In Seed & Series A, B, C Funding Rounds
- Total Funding Ranges From $50k In Pre-Seed To $3.5 Billion In Series H
Want The Thousands Of Raw Data Of Ex-Google Employees?

Methodology

At Proxycurl, we enrich only public LinkedIn profiles (public profiles can be scraped with legal precedence) & from this we identified 117,981 Google employees.
- There were however 7,872 profiles that returned 404 error with unbeknowst reasons: invalid profiles, private LinkedIn profiles, changed usernames, deleted LinkedIn profiles etc.
Out of which, 2,762 people indicated a change of employment in the last 4 months (from the date of writing i.e. 14 February, yes Valentines’ Day) - aka ex-Googlers that got laid off.
- Why only 2,762 and not the 12,000 as announced by CEO Sundar Pichai? A main reason could be they have yet to update their LinkedIn profiles. Also, the announced 12,000 could include blue-collar workforce who are not on LinkedIn. Proxycurl dataset is also focused on English-speaking regions, thus would not cover extensively some regions such as India.
Hereon, the rest of our analysis will focus on the remaining valid profiles. Note: even then some data fields might be absent if the individuals and companies did not fill in the particular fields in LinkedIn, thus certain parts of the analysis might leave out these profiles.
The 4 main Proxycurl products that we used for this piece are:
1. Using the Employee Listing API Endpoint, we fetched a list of all past & present employees of Google.
2. From the list of “Present employees,” we used the People Profile API Endpoint to refresh their profiles to identify who have landed a new job.
3. Using the Company Profile API Endpoint we then update the profiles of these new companies they joined to get more information.
4. Our LinkDB LinkedIn Database was also used to identify historical data relating to their time in Google.

Now, let’s dive into the data insight proper.

Part I: People Profiles Of 2,762 Ex-Google Employees - Who Are They?

Who are they? We first take a look at how the layoffs have impacted them, and who it has impacted, namely which countries were most impacted, and which roles?

Every 2 Laid-Off Googlers, >1 Is From The States (56%)

Top Impacted Countries
Naturally, most of the impacted Google employees were based in the States as Google employed the majority of its 117,981 employees from the States and the States being its HQ. Therefore the US herself saw 1,158 (or 56.4%) Googlers leaving the company.

The next countries did not even come close - with United Kingdom in second at only 3.3% of affected Google employees, Ireland with 2.8%, India and Canada with 2.3% and 1.7% respectively. These 4 countries only make up the next 10% of ex-Googlers after the US' 56.4%.

Notably Ireland, and Singapore which came next in 6th made the top despite their small country population (about 5 million respectively), a testament to Google’s human resource investment there. Or de-investment post-layoffs.

Top Impacted Region
Region wise - discounting North America due to the US’s heavy weightage (and Canada topping 5th) - Europe saw 12% of ex-Googlers let go and Asia with 5%, rounding up the 2nd & 3rd most regions with ex-Googlers globally.

Below, the US is excluded from the chart above to see the impact on other countries.

Impact On US States
Note: 44% of the 1,558 US profiles did not return states data, so the analysis will be based only on the remaining 56%.

With that, California being Google global HQ had a whooping 30% of total ex-Googlers (out of the total universe of 2,762), that is quite a lot of people laid off from a single location. The next 3 states are New York at 8.8%, Washington 5.5%, Texas 3.3% (coincidentally they are palindromic decimal). These 3 states themselves have more Googlers laid off then each country that came after the US (UK, Ireland, India).

Food for thought: with US having most of the laid-off Google employees, this inadvertently led to a significant number of them joining the US's huge startup scene - with the country having by far the highest startup investment globally.

Software Engineers Were Most Affected: 30% Of All Laid-Off Employees

Casual observers often remarked that the most affected role is the software engineers. Well, we found that it is indeed true. Out of all 2,762 ex-Google employees, 30.6% (or 844) people were ex-Google software engineers.

As the title field in LinkedIn is a free text input, there are more than 1,500 distinct roles that we observed from our data collected. We then used ChatGPT to clean up and categorize all 2,000+ roles into 9 main categories. Finally we did a quick manual comb-through to correct any discrepancy.

Coming in 2nd to 5th after SWEs are these categories: sales, recruitment, product design, and project management, each with around 11-12% of ex-Googlers each. These few categories combined with software engineers would have amounted to 77% of total roles laid off in Google.

Food for thought: Being part of Google and now laid off, was that what spurred these SWEs to bring their game to the startup scene for a change?

Part II: 747 Ex-Googlers Found New Jobs - At Which Companies?

Now, what’s next for these ex-Googlers? Let’s take a look at where they have headed next, have they all found the next destination, some have founded their own startups - nothing unusual of course from these ex-Googlers.

747 Ex-Googlers Landed New Jobs / Founded New Companies

747 out of the 2,762 laid-off Googlers have since updated their profiles to indicate a change of employment status in the last 4 months since November 2022.

There could of course still be many that have found new jobs, which they might not yet update their profiles - a common practice on LinkedIn. Due to reasons such as: they simply do not need to do so upon securing a new job, they forgot, they wanted to keep their new movement private for a few months, and so on.

58% Of New Companies Are In Technology & IT, Includes Software Development

From our data collected, there were a total of 103 distinct industries identified, from 707 valid company profiles out of 747 ex-Googlers who indicated new employment.

With these 103 distinct industries, we then ask our beloved ChatGPT again to categorize them into a few main categories. It did a pretty good job - discrepancies in some smaller categories aside which we needed to manually clean up.

A total of 56.7% (or 401) companies that ex-Googlers joined are in the ”Technology & IT” industry, out of which 65.6% (or 263) are in “Software Development.” This is well expected because of the people being from Google. If we were to look at the entire pie of 707 new companies, “Software Development” alone takes up 37%.

Food for thought: among these 401 "Technology & IT" companies are startups collectively worth more than $15 Billion that ex-Googlers now join.

Coming in second is the category of “Business & Professional Services“ at 24.2% of total (or 171 companies). This category includes - quoting the top few industries - financial services, business consulting, advertising services and recruitment.

A Sample Of Companies That Hired Ex-Googlers

Company	No. of Ex-Googlers Hired
Stealth Startup	13
Moloco	3
Arta Finance	2
Unlikely AI	1
AgileRank	1
VAST Data	1
Samsara	1
Optimalex	1
Yonder	1
Flock Safety	1
Time for the Planet	1
Kitman Labs	1
Isomorphic Labs	1
And 694 other companies. Email us for the exhaustive Google Layoff Dataset.

Part III: Billions Worth Of Impact Made By Just One Third Of Ex-Googlers

In this section, we want to understand how the departure of this many ex-Googlers impacts the startup ecosystem. What kind of industries and startups welcomed most of these ex-Googlers?
Methodology: we identified startups as companies that fulfill any one of these parameters:

Received funding
<=50 employees
Ex-Googlers joining as "founder" or "co-founder"
Incorporated from 2020 onwards

One-Third (or 205) Ex-Googlers Joined Startups

With the parameters above, there are 205 startups/ventures identified. With further breakdown, the list below shows the proportion of companies that fulfill each of the parameters, with a deep dive for the fundings information in the next section.

73 Companies Are Funded: these companies indicated their latest funding rounds being either of these:
- Pre-seed, seed, series A to H and convertible note.
121 Companies Have <=50 Employees, these are the breakdown:
- 82 companies indicated 50 employees,
- 37 companies have 10 employees,
- 2 companies have 1 sole employee.
43 Companies Have Ex-Googlers As Founder/Co-Founder:
- 14 are founders,
- 23 co-founders, consisting of CEOs, CTOs & CROs,
- 6 founding members (engineers, designers etc).
41 Are Incorporated From 2020:
- Of those 13 companies founded in 2020, 3 have reached 200 employees, 2 have 500 employees.
- Those 18 founded in 2021, 3 has 200 employees, 1 has 500.
- For the 10 founded last year (2022), half have 10 employees and the other half have 50.

Startups Recently Founded By Googlers Who Were Recently Laid Off

Company	No. of Ex-Googler Founders
Stealth Startup	11
Vizcom	1
TechShack	1
Stealth Conversational AI Startup	1
Plato	1
Pigeon (YC W23)	1
And 13 other startups. Email us for the exhaustive Google Layoff Dataset.

Equal Proportion Of Startups In The Same Industries As All-Company Universe - 55% In Technology & IT

Similar to the total universe of 747 companies ex-Googlers joined, companies in the "Technology & IT" category take up also 54.5% (103) of the 205 startups. Out of which, "Software Development" takes up a slightly higher percentage than total company, at 43% (or 81).

Ranked second is also the "Business & Professional Services" category, at a slightly higher of 31.2% out of total startups, and with the same few top industries.

Close To 70% Of Ventures Are In Seed & Series A, B, C Funding Rounds

Majority of latest funding rounds indicated in these company profiles are in the earlier stages, with the later stages tapering off to 6 and 1 companies.

These earlier stages accounted for more than 68.9% of the funding rounds indicated.

On the other end of the spectrum, pre-seed has 3 companies stating that as their last funding round.

Total Funding Ranges From $50k In Pre-Seed To $3.5 Billion In Series H

Note: the chart below has funding amount rounded to the nearest $100,000 for easy reading.

The pre-seed round saw the lowest 2 funding amounts of $50k and $75k. Next lowest is $380k in Seed round, and from there on the funding amounts increase gradually in the subsequent rounds. Series A started from $4.9 Million and capped at slightly below $100 Million.

Series B & C both have a range of below & above $100 Million. Series D & E however have all $100 Million and above. In Series F one company first crossed the billion-funding amount at $2 Billion. For Series G, data wasn’t available in LinkedIn for our scraping.

Total fundings from all companies amounted to $14,956,105,000.

Want The Thousands Of Raw Data Of Ex-Google Employees For Your Own Applications?

There you go! Some analysis of ex-Googlers’ latest movements, as of the time of data collection and writing. Of course our data analysis processes might not be top notch and while we did not commit to achieving that, our commitment was towards the quality of data that we scraped - being fresh as we scraped the data live for this analysis, accurate, and fully-legal.

It is certainly unfortunate that so many people’s roles were affected in these headline-layoffs. There are certainly many who capitalized on opportunities to do something they’ve always liked, founded companies and so on.

Nevertheless if you like what we have put together, we have something tasty for you:

Free data pack with more than USD $11,300++ worth of structured data of ALL ex-Googlers that we talked all about in this piece.

This valuable data pack is available only to VC firms and people in the media industry (i.e. journalists, editors, publishers etc). What you need to do is to just drop us an email at hello@nubela.co and we will get in touch. Get a copy yourself and you can deep dive into the data even more.

Subscribe to our newsletter or Twitter to get notified when the next in this series is out: Amazon *(with Meta, Netflix and more coming up next)*. Cheers.

Best Tech Affiliate Program (2023) – Refer Once & Earn for 2 Years

Joseph Lim — Fri, 19 May 2023 04:06:14 +0000

Have you heard of an affiliate program where you can continue to earn from each referral for two years after they sign up? Well, you've come to the right place: We offer such a program here at Proxycurl! And our commission is a lucrative 10% of your referrals' spend throughout these two years.

If you have an established platform - or multiple platforms - with a tech audience, keep reading to learn more about what Proxycurl is and the structure of our affiliate program.

Know what you’ll be promoting - what is Proxycurl?

Before we expand further on the affiliate program, let us tell you more about Proxycurl and our products.

At Proxycurl, we offer two primary products:

Our suite of API endpoints enriches person and company profiles with structured data such as emails and phone numbers, job and salary info, etc. Meanwhile, our LinkDB product is an exhaustive dataset of more than 470 million LinkedIn profiles globally. We have helped thousands of businesses of all sizes and from all industries globally, and we're sure that our products will interest your audience too.

Feel free to check out our website for more information about our company and products, especially our blog and FAQ pages.

Who is best suited for our affiliate program?

Our affiliate program is ideal for you if you are:

Someone with an established tech audience,
Building up your network, or
Intending to venture into the tech industry.

This profile includes tech bloggers, writers, tech influencers, and so on.

Furthermore, our affiliate program is ideal for your if your audience contains software endineers or data engineers, especially at:

Of course, if you have someone in mind who you think might have great use case in mind for our product, don't hesitate to pitch them directly.

Superb commission structure and referral period

We understand that the key to any affiliate program is, well, you. We offer a fixed 10% commission on all purchases made by your referrals for an impressive period of two years. This means that you'll continue to earn commissions on every purchase made by any referral for the first two years of their initial sign-up.

In addition, tracking your commission is easy. All you need to do is share your unique UTM links with your audience, and any purchase made by your referrals will be immediately reflected in a dashboard.

Join Proxycurl Affiliates & start earning commissions

If you like what you've heard, join Proxycurl Affiliates right now and start earning lucrative commissions today. With our powerful suite of products, highly satisfied user base, and an incredible commission structure, you'll have no trouble earning on our platform.

If you have any questions, email us at marketing@nubela.co If not, visit our dedicated affiliate page right now to sign up and start earning!

How to Build a LinkedIn Data Scraper

Joseph Lim — Tue, 09 May 2023 08:58:09 +0000

So you want to scrape LinkedIn for data. Maybe you're a dev looking to automate your next remote job search; maybe you're a hiring manager trying to find the perfect candidate; or maybe you're an investor trying to figure out who exactly is joining which stealth ventures. The problem, of course, is that LinkedIn doesn't have a public API, and building your own scraper from scratch is really hard.

In this guide, we'll go over the DIY solution in case you want to try it yourself despite the difficulty. And after that, I'll walk you through the Proxycurl API, a SaaS product built precisely to solve the problem you have right now, using Postman, Python, and Javascript. If you're here for an easy solution, feel free to skip to the "answer."

Here's what we'll cover:

What's required to scrape LinkedIn yourself? Understanding private vs public profiles, scraping HTML, bypassing the authwall, and more.
What does the Proxycurl API accept, and what do you get?
How to query the Person Profile Endpoint in Postman, Python, and Javascript.
A measure of reliability: We'll query the Person Profile Endpoint 100 times and look at the status codes. ## How would you build a LinkedIn scraper in a vacuum?

First, we must understand the difference between public and private profiles on LinkedIn. In a nutshell, it's only safe and legal to scrape public profiles. Companies that scrape private profiles have faced lawsuits from LinkedIn. On the other hand, public profiles are legal to scrape.

So if we limit ourselves to private profiles we will be okay. But how do we go about actually scraping a profile?

Scraping one field from an example profile

For this article, we saved an example profile locally and wrote a simple scraper to parse out one of the easier fields to retrieve as JSON data: A person's languages. Note that this script is doing nothing to connect to LinkedIn; all it's doing is using BeautifulSoup to scrape one field. Here's the code:

def get_languages(raw_html: BeautifulSoup) -> List[str]:
    languages = []

    language_section = [_ for _ in raw_html.body.main.find_all('section') if _.get('data-section') == 'languages']

    if len(language_section) == 0:
        return languages

    for language in language_section[0].div.ul.find_all('li'):
        languages.append(language.div.h3.get_text().strip())
    return languages

Depending on your level of experience with web scraping, this may seem like a lot of work for just a single field, or it may not seem too bad. Either way, keep in mind that we also have to:

Do this for every single field we're interested in.
Keep up with a moving target. Any time LinkedIn rearranges any of its HTML, we'll have to adjust our parser, sometimes significantly, and sometimes all at once.
Support internationalization. For example, Czech-language profiles need a dedicated parser.

Authwall: What is it and how do we deal with it?

Go into incognito mode and try browsing some LinkedIn profiles. Go on, I'll wait. Chances are, within one or two clicks you arrived at something that looks like this:

Figure: LinkedIn's "authwall" blocks you from seeing most profiles while you're logged out.

That's the "authwall." Even if a LinkedIn profile is public, you still need to be logged in if you want to see it. We can access it in code like this:

import requests
from bs4 import BeautifulSoup


def has_authwall(url: str) -> bool:
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    for s in soup.find_all('script'):
        if '/authwall?trk=' in s.get_text():
            return True
    return False

(Note that we're still using BeautifulSoup here instead of raw searching for the string in case someone decided to troll us by including the raw string /authwall?trk= in their profile. Hey, it could happen.)

Although this is how to detect the authwall, actually bypassing it is a lot more complicated and out of the scope of this article; we aren't going to share how Proxycurl does it (sorry!).

Scraping LinkedIn yourself is hard, but scraping LinkedIn doesn't have to be

As you may be concluding, it's really hard to DIY a LinkedIn scraper. That's why Proxycurl exists. We take care of everything for you, and you can take advantage of any one of our many endpoints to retrieve all the data you could possibly want in JSON format.

What do I put into the Proxycurl API, and what do I get out?

What you have

Proxycurl is a paid API, and you need to authenticate every request with a bearer token (your API key). After you've signed up for a Proxycurl account, you can retrieve your API key from the dashboard. Signing up gives you 10 credits, and you'll get another 5 credits upon your first successful query of the Person Lookup API Endpoint, so you can experiment a bit.

If you've been paying close attention, you might have noticed I've now said two slightly different things - I mentioned the Person Profile Endpoint in the introduction, and now here's the Person Lookup Endpoint. What's that all about? Yes, these are different endpoints, and yes, they give you different things. The Proxycurl docs have an ELI5 listing all of our endpoints that gives you a what-you-have-what-you-get directory so you can figure out which one to use. And you thought address books were outdated!

For now, though, let's concern ourselves with the Person Profile Endpoint, this one's just a bit shinier to show off in a tutorial. You can find this endpoint nested under the People API. It assumes you have a LinkedIn URL and you want to get a bunch of information about that person.

What you get

What you get will be a response in JSON format. No parsing of any sort will be needed. Even timestamps are given to you as objects, so you can feed them to datetime without doing any string parsing.

Exactly how much is that "bunch" of information we mentioned above? It depends on how much any given person has made available on their profile. First of all, the user needs to have made their profile public in the first place; private profiles will 404. You will never be charged for a request that returns an error code.

You can see the full list of standard fields if you scroll a bit down in the Proxycurl Person Profile API docs, but some you might be particularly interested in include experiences, languages, city, state, volunteer_work, people_also_viewed (a list of similar profiles), and other such fields. You can see a full example of any response in the documentation on the right-hand side below the request that generated it.

There's a bit more, though. Proxycurl can attempt to go above and beyond, and scrape the following fields if you request them:

skills
inferred_salary
personal_email
personal_contact_number
twitter_profile_id
facebook_profile_id
github_profile_id
extra

Each of these additional fields that you request comes at an additional credit cost, so don't request them unless you require them. But when you do need them, Proxycurl puts them a single parameter away!

Example walkthroughs

Now that we know what we're working with, let's look at some examples. We'll start with Postman and Python, since we can pretty much copy-paste these from the Proxycurl docs, and then we'll write a bit of JS code.

Postman

We've previously posted an in-depth Postman walkthrough, but you can follow along here too, and refer to that article if you get stuck.

In the Person Profile Endpoint docs, find the orange button that says Run in Postman and click it. Then click "Fork Collection" and log in however you like. If you're logging in for the first time, the screen should look something like this:

Figure: Postman upon initially forking the Proxycurl collection

Add your Bearer Token to the environment as a variable. Limit it to Proxycurl. You can do this from the Variables tab or from the pop-up that appears when you start typing into the "Token" field. You can name this variable anything you like, but Bearer Token is a good name.
Verify that the Authorization type is set to "Bearer Token" and that you have typed {{Bearer Token}} into the Token field and click Save in the upper right-hand corner. Remember to click Save!! Your page should look like this:

Figure: Adding a Bearer Token variable Authorization in Postman

Now, we should be able to run something! On the left-hand side, under "My workspace," expand your Proxycurl collection and then the People API, and double-click the Person Profile Endpoint. You can un-check some of the fields if you want or modify others - for example, you might want to change use_cache from if-present to if-recent to get the most up-to-date info, but maybe you don't need the user's personal contact number this time around.
Once you've modified the fields to your liking, click the blue "Send" button in the upper left-hand corner.

Troubleshooting Postman

If you get a 401 status code, most likely you forgot to hit Save in Step 3. A good way to troubleshoot this is to see if you can fix it by editing the Authorization tab for this specific query to be the {{Bearer Token}} variable. If that fixes it, then the auth inheritance isn't working, which probably means you forgot to save.

Python

Let's now try and make the exact same request, only with Python. In the Proxycurl docs, there's a toggle at the top of the page where we can switch between shell and Python. Here is the right-hand side only of the Person Profile Endpoint section, see the upper-left-hand corner of this screenshot:

In that screenshot, we can simply paste in our API key where it says YOUR_API_KEY. However, it's best practice to be just a little more careful with secrets than that when you regularly publish your Python code on the internet so we'll do slightly more work. Also, we'll extract the JSON response at the end and print it. Here's the code for all that, and feel free to use this yourself:

import os, requests

api_endpoint = 'https://nubela.co/proxycurl/api/v2/linkedin'
api_key = os.environ['PROXYCURL_API_KEY']
header_dic = {'Authorization': 'Bearer ' + api_key}
params = {
    'url': 'https://www.linkedin.com/in/johnrmarty/',
    'fallback_to_cache': 'on-error',
    'use_cache': 'if-present',
    'skills': 'include',
    'inferred_salary': 'include',
    'personal_email': 'include',
    'personal_contact_number': 'include',
    'twitter_profile_id': 'include',
    'facebook_profile_id': 'include',
    'github_profile_id': 'include',
    'extra': 'include',
}
response = requests.get(api_endpoint, params=params, headers=header_dic)
print(response.json())

What we get is a huge wall of text in our terminal. But what if we could, say, put a breakpoint and inspect the actual output using PyCharm? Let's give it a try. Change the last two lines to this:

response = requests.get(api_endpoint, params=params, headers=header_dic)
result = response.json()
print(result)

And then put a breakpoint on the print statement, and run it with the debugger. And then we can drill down into every field of the result JSON, like so:

Fantastic! The debugger is an amazing tool, even when there are no bugs in your code and you just want to see the shape of your data.

JavaScript

The following code is meant to be pasted into your browser (i.e. it's not Node.js code). To avoid CORS errors, paste it into a page on the Proxycurl domain, for example the docs page. IMPORTANT: Never paste code into your browser console that you don't understand. This is a major security issue! Ensure you fully understand any JS code you are executing before running it.

Here is an overview of what the code below is doing:

Generate an object of params corresponding to what we want to fetch in our query.
Transform these params into a query string using URLSearchParams.
Make the request using fetch, including the bearer token as a header.
Catch any errors and otherwise print a JSON to the console.

And here is the code:

const apiKey = "your_api_key_here";
const params = {
  url: "https://www.linkedin.com/in/johnrmarty/",
  fallback_to_cache: "on-error",
  use_cache: "if-present",
  skills: "include",
  inferred_salary: "include",
  personal_email: "include",
  personal_contact_number: "include",
  twitter_profile_id: "include",
  facebook_profile_id: "include",
  github_profile_id: "include",
  extra: "include",
};

const queryString = new URLSearchParams(params).toString();
const endpoint = `https://nubela.co/proxycurl/api/v2/linkedin?${queryString}`;

fetch(endpoint, {
  headers: { Authorization: `Bearer ${apiKey}` },
})
  .then((response) => {
    if (!response.ok) {
      throw new Error(`Network response not ok. Status code: ${response.status}.`);
    }
    return response.json();
  })
  .then((data) => console.log(data))
  .catch((error) => console.error("Error: " + error));

How reliable is Proxycurl?

Finally, let's take a look at reliability. We'll do this in Python. Reusing the same code from above, let's loop the request 100 times:

codes = {}

for _ in range(100):

    response = requests.get(api_endpoint, params=params, headers=header_dic)
    code = response.status_code
    if code not in codes:
        codes[code] = 1
    else:
        codes[code] += 1

print(codes)

Do any of them fail? Here's the output:

{200: 100}

In other words, all of them were successful. In fact, last week when I was writing another post called Using Proxycurl's Historic LinkedIn Employee Count Tool for Investment Research I was regularly bursting 1500 queries at a time, and I found the API completely reliable, reproducing my results each time I ran the same query. If you enjoyed this article, do check that one out - it shows off a pretty cool application of this API.

Summary

In this article, we first discussed how to make a DIY LinkedIn scraper. Then we went over how to build a LinkedIn data scraper easily with Proxycurl using any of the following three methods:

Postman
Python
JavaScript

We introduced you to the Proxycurl Person Profile Endpoint, which returns JSON data to you, so that you don't have to do any text parsing of any kind upon receiving your response. For an overview of the rest of Proxycurl's endpoints, check out the what-you-have-what-you-get ELI5 in the Proxycurl documentation. Now that you know how to use one endpoint, you know how to use them all!

We'd love to hear from you! If you build something cool with our API, let us know at hello@nubela.co! And if you found this guide useful, there's more where it came from - sign up for our newsletter!