DEV Community: dnasedkina

Navigating the Data-Sourcing Landscape: Trends, Proxy Providers, and Future Prospects

dnasedkina — Wed, 14 Jun 2023 12:32:42 +0000

The data-sourcing industry has experienced significant developments in recent years, with an increasing availability and demand for various data sets. One notable trend that emerged around four to five years ago is the rise of proxy providers and companies offering APIs for automatic website extraction. This trend has gained momentum, leading to numerous companies launching API structures to meet the growing needs of data extraction.
In the Ethical Data, Explained podcast by SOAX, host Henry Ng is joined by Pierluigi Vinciguerra, the co-founder and CTO at Re Analytics, who has spent ten years creating web scrapers and maintaining web scraping knowledge. Together, they discuss how to choose the right proxy provider, understand client preferences, and the dynamics of data sets.

Choosing the Right Proxy Providers

One critical aspect of assessing data sets is the sourcing of IP addresses from proxy providers. This is particularly important for businesses operating in highly regulated industries, such as hedge funds. As hedge funds are subject to strict regulations, it is essential to ensure that the information gathered through proxies is obtained legally and without any ambiguity. Maintaining transparency and adhering to regulations is crucial in such cases.

Apart from legality, pricing plays a significant role. Operating at scale requires cost-effective solutions, making pricing a key consideration for businesses. Additionally, the size of the IP pool offered by the provider is also essential. Some large websites employ IP blocking, which can pose a challenge to data gathering. To overcome this, having a vast pool of IPs allows for comprehensive data collection without disruptions. It is evident that businesses must consider multiple factors when selecting a proxy provider. Apart from legal compliance, factors such as pricing and IP pool size are crucial in ensuring the smooth and uninterrupted gathering of data from various sources.

Key Factors for a Successful Web Scraping Project

Assessing the nature and demand of data sets requires a comprehensive approach. The significance of this task cannot be underestimated, especially when considering smaller projects. In such cases, the foremost concern lies in the quality of the data output. Whether the project is being presented internally within one's own company or to an external entity, establishing a foundation of trust between the provider and the user becomes paramount. Any inaccuracies or discrepancies in the data can lead to the erosion of this trust. Consequently, it becomes essential to invest substantial effort in ensuring the delivery of accurate and reliable data.

To achieve this objective, implementing a robust process for data quality becomes imperative. Several techniques can be employed, including human count, regression analysis, trend analysis, forecasting, and other applicable methods. These techniques serve as vital tools in maintaining the integrity and precision of the data. While these practices hold true for both small and large-scale projects, the latter necessitates additional considerations.

In large-scale endeavors, the significance of data quality remains high, but achieving it requires a broader perspective. Attention must be given to the underlying architecture of the scraping process. When building a scalable system, standardizing the processes and logs becomes crucial. The overall architecture plays a pivotal role in the effectiveness and efficiency of the data collection and analysis. Therefore, a meticulous approach is required in designing the architecture, accounting for the specific demands of large-scale projects.

Emerging Trends and Transformations in Data Extraction

Amidst the proliferation of APIs and extractors offered by proxy providers, attention has shifted towards the ethical sourcing of IP addresses. Many providers now emphasize the importance of responsible and legitimate web scraping practices. This shift is pivotal in changing the perception of web scraping from a shady industry to a legitimate and valuable means of data acquisition.

With data scraping and proxies becoming more accessible to businesses of all sizes, the discussion turns to the future of data extraction in the next decade. Pierluigi acknowledges the growing interest in web data from companies but notes the increasing cost associated with web scraping projects. Currently, only large companies can afford extensive web scraping endeavors, while smaller companies may struggle to achieve satisfactory results with in-house scraping due to a lack of expertise.

“At the moment only big companies can afford a great web scraping project. But if you are a small company trying to make the web scraping project in-house. The results will be mixed because they don't have the skill to do it. That's why we started Databoutique because we wanted to sell prescriptive data at a low price for everyone”
Pierluigi Vinciguerra

In response to this mismatch in demand and affordability, Databoutique enters the picture. Databoutique aims to bridge the gap by offering prescriptive data at affordable prices, catering to companies that seek reliable and cost-effective data solutions.

The future direction of data extraction could go either way. Some companies may choose to develop their in-house solutions, relying on advanced skills and expertise, while others may opt for platforms like Databoutique to meet their data needs. However, the existence of a slight gap between the current capabilities of businesses and their desired future state suggests room for further progress and development in the industry.

To find out more, tune in to the full of this week’s episode! You can find links to Apple podcasts, Google podcasts, Spotify, and YouTube on the official podcast webpage - Ethical Data, Explained!

The Power of Privacy-Oriented Tools: Balancing Personalization and User Privacy

dnasedkina — Thu, 18 May 2023 09:06:04 +0000

In today's digital landscape, the significance of the browser we use often goes overlooked. Your browser serves as the gateway to the vast realm of the internet, shaping your online experiences and managing an immense amount of sensitive personal data.

Many people don't give much thought to the default browser that comes pre-installed on their laptop or smartphone, whether it's Safari, Microsoft Edge, or any other commonly used option. It's easy to assume that this default choice is the best or only option available.

However, the reality is quite different – a unique browser’s fingerprints can identify individuals even in incognito mode. Here's another example: a certain amount of data leakage is going from the search engine to the third-party provider, especially if you clicked on an ad.

So, can next-gen browsers restrict your IP address and help eliminate your fingerprint? Will a VPN or proxy keep you private? And can you layer these tools together for an additional level of privacy?

In this episode of the Ethical Data, Explained! podcast by SOAX, host Henry Ng is joined by Michael Levit, co-founder and CEO of Tempest. They will discuss privacy-oriented tools that can help restrict the amount of information leaked while browsing the Internet, and the different layers of protection available. They will also emphasize the importance of informing users about ad tracking, fingerprinting, and callbacks that record every click, and ways to give users control over their privacy, whether they choose to be completely private or not.

The Connection Between Search Engines and Browsers

In recent years, we have seen numerous examples of companies collecting and misusing personal data, leading to increased concerns about online privacy. In response, privacy-oriented search engines and web browsers have emerged as a solution to this problem.

Search engines and web browsers are closely related, and many users do not differentiate between the two. Traditional search engines like Google prioritize personalization, collecting vast amounts of data about users to tailor search results.

We took Chrome and then we ripped out all those pieces that capture every click. When you remove those components, our browser is faster than Chrome.
Michael Levit

In contrast, privacy-oriented search engines like DuckDuckGo focus on user privacy by offering private search capabilities. However, building a browser that encapsulates both a great private experience and great functionality is challenging. Therefore, both a search engine and a browser are needed to achieve this balance.

The Importance of User Choice

One of the most important aspects of privacy-oriented tools is that they empower users to take control of their online privacy. Many users are just beginning to realize that they have a choice in web browsers and can download and install different ones than the default option that comes with their computer. By choosing a privacy-oriented browser, users can ensure that their data remains secure and private.

Features of Privacy-Oriented Tools: The Power of VPN and Proxy Technology

The use of privacy-oriented search engines and web browsers provides users with additional features such as the ability to search without cookies, block tracking scripts, and encrypt search queries. These features maintain the privacy and security of users' online activity and prevent the collection and use of their information without their explicit consent.

Furthermore, VPN and proxy technology can be used alongside privacy-oriented search engines and web browsers to provide an additional layer of protection, further enhancing online privacy.

To protect yourself from a privacy standpoint online, a browser and a proxy are two excellent places to start. If you are not doing anything else, do those two things.
Michael Levit

By integrating VPN and proxy technology directly into the browser, these tools make it even easier for users to protect their personal data.

Striking a Balance Between Personalization and Privacy

While privacy-oriented search engines and web browsers prioritize user privacy, they must also strike a balance between privacy and personalization.

The moment I'm sharing information between my devices, I'm giving the browser information about me. So we have created universal identifiers that are unique to you but know nothing about you.
Michael Levit

Users should have the option to choose whether or not to share their data in exchange for accessing certain websites or services. For example, sharing passwords may require sacrificing privacy to enjoy sharing content with friends on platforms like Facebook.

Privacy-oriented tools can allow for targeted advertising based on the keywords a user is searching for at a given time, supporting business models without collecting excessive user data.

Tempest has no memory of advertising, and only shares the general search string with Microsoft, their only ad partner, to stay true to the principles and ideologies that the company was founded upon.

Tips for Enhancing Online Privacy: Browsers, Proxies, and Search Engines

By following three simple steps, you can take control of your data and enhance your online privacy and security:

Get a browser: Begin by downloading and installing a browser of your preference, with a default search behavior. This can impact your information and data going forward, so it's important to make a thoughtful choice.
Install a proxy: Consider using a virtual private network (VPN) or proxy server to encrypt your internet connection and protect your online activity from prying eyes.
Choose a search engine: Change a default search engine to a search engine that doesn't track your online activity. This way, you can search the web without leaving a trail of data behind.

That’s it for today. To find out more, tune in to the full of this week’s episode! You can find links to Apple podcasts, Google podcasts, Spotify, and YouTube on the official podcast webpage - Ethical Data, Explained!

Unleashing the Potential: Bridging the Gap between On-Site Marketing and Big Platform Advertising

dnasedkina — Wed, 17 May 2023 07:23:48 +0000

In today's digital landscape, the global online advertising industry has achieved unprecedented heights, with an astonishing expenditure of approximately $600 billion. This colossal figure underscores the magnitude of a market primarily driven by brands seeking to acquire traffic through platforms such as Facebook, Google, TikTok, and others. These brands heavily invest in these platforms to target potential customers and direct them to their websites, allocating less than 5% of their budget to marketing, engagement, and expanding reach on their websites.

It is evident that on-site marketing efforts significantly lag behind the robust strategies deployed on these prominent platforms. While these platforms undeniably excel in equipping businesses with powerful tools for efficient operations and effective communication, the recent implementation of cookie laws has hindered precise audience targeting, resulting in decreased efficiency.

In this episode of the Ethical Data, Explained podcast by SOAX, host Henry Ng is joined by Romain Vermeulen, the head of data science at Wisepops. Together, they delve into the realm of different methods that assist brands in engaging with visitors.

Seizing the Right Moment: Wisepops' Decentralized Mission

To address this discrepancy, Wisepops, a trailblazer in the field, aims to empower brands and bridge the gap between on-site marketing and big platform advertising. By offering a range of on-site channels, including notifications, pop-ups, and other engaging formats, Wisepops helps brands understand visitor preferences, effectively convey brand messages, and ultimately drive conversions.

The timing is crucial, as recent privacy laws have reduced the efficiency of targeting on large platforms. Brands are now paying more for diminishing results, making it essential to shift focus to the post-landing stage of the customer journey—the right side of the funnel. Recognizing this shift, Wisepops emerges as a pivotal solution, enabling brands to establish direct connections with visitors on their websites and fostering meaningful engagement that drives conversions.

“I was particularly attracted by the decentralization nature of the mission of Wisepops, which is to empower brands to engage with their audience on their own website.”
Romain Vermeulen

By empowering brands of all sizes, Wisepops aims to become a comprehensive marketing platform, offering intelligent products, multiple channels, and easy-to-use automated tools and campaigns. The company strives to transcend being a mere solution and instead transform into a dynamic marketing platform capable of empowering brands to thrive.

Privacy-Centric Data Collection: Enhancing Visitor Experiences

Privacy regulations are at the forefront of Wisepops' practices. Every step revolves around obtaining visitor consent before initiating data collection. Through a well-defined process, Wisepops ensures compliance with GDPR and other privacy regulations. Anonymous tracking is conducted on the browser level, respecting visitors' consent to cookies and privacy policies. Valuable insights into visitor behavior, such as device usage, geographic location, and product preferences, are acquired, enabling the delivery of personalized experiences.

Challenges in Data Collection and Privacy Compliance

Adhering to privacy regulations presents additional complexities for Wisepops, necessitating meticulous attention to data collection practices. The team takes extensive measures to guarantee the anonymity of collected data, implementing encryption during data transit and storage. With a streamlined flow and a focus on a secure design, Wisepops prioritizes security and mitigates potential vulnerabilities.

The Future of Wisepops: Growth, Innovation, and the Role of ML and AI

As technology continues to advance, the company will leverage machine learning and AI to stay at the forefront of on-site marketing.

The company aims to tap into the on-site marketing market's immense size and address prominent pain points. For instance, Wisepops aims to tackle cart abandonment by developing smart segments that predict and engage potential customers effectively. Additionally, by optimizing on-site notifications and leveraging deep content capabilities, Wisepops intends to maximize engagement, particularly on mobile devices.

With substantial investments in its tech infrastructure and data stack, Wisepops has set a solid foundation for future growth. The company is prepared to develop and deploy intelligent use cases, providing more effective and intelligent campaigns on customers' websites. With numerous opportunities waiting to be explored, Wisepops is committed to making a significant impact and delivering value to its customers.

Ethical Data Handling: Protecting Your Brand

dnasedkina — Tue, 16 May 2023 11:44:20 +0000

On this episode of Ethical Data, Explained by SOAX a project manager, tech, and lead solutions architect, but currently, the head of Technology at Brand Guard startup Adolfo Hoyos joins the discussions with Henry Ng about the common mistakes in e-commerce data handling and tips and tricks to protect brand reputation.

Avoiding the Common Mistakes in E-commerce Data Handling

In terms of storage and handling of data, the key mistakes that e-commerce sites make are:

Taking private data from users.
Making that information visible to other users in any way.
Sharing overall metrics about customers with other users.
Hiding data behind terms and conditions.

Expert Tips for Overcoming Malicious Actors, MAP Violators, and Gray Market Threats

The term "malicious" is subjective and what may be considered malicious by one party may be benevolent to another. Thus, the Brand Guard that eliminates undesired sellers from marketplaces may be seen as malicious by the sellers but is seen as necessary by the company to protect the brand and enforce policies.

Therefore, there are some general tips to help minimize the damage from malicious actors and other online threats:

Be cautious about what information you post online, especially on public websites.
Assume that everything you post online could be public and accessible to anyone.
Be careful about what you upload.

With resellers, such as map violators and gray market sellers other issues can arise. The severity of these issues depends on the brand and the strength of its map policy and supply chain management. When brands have strong policies and enforce them with all sellers and resellers, they can prevent supply chain leaks and maintain control over their map price and product quality. Authorized sellers can also provide warranties and easy returns for customers, which can be affected by supply chain leaks. Overall, having strong policies in place can help brands maintain control over their products and pricing.

There are some tips for a company that helps brands build reseller policies and strategies to combat gray market sellers:

Draft agreements with resellers and distributors to establish clear expectations and guidelines for how products can be sold and distributed. By having a written agreement in place, the company can ensure that its resellers and distributors understand and adhere to these policies.
Web scraping tools are not an ethical problem as long as the data being gathered is public and not behind a specific access for a user. So, the utilization of AI and web scraping automation tools to check for possible Minimum Advertised Price (MAP) violators or unauthorized sellers help to maintain the authorized seller network.
Multiple access levels, subdividing access levels into teams, and providing users with access levels ensure that user data are managed responsibly.

For more info on Adolfo Hoyos and the story behind Brand Guard, welcome to the official podcast webpage.

The Ethics and Legality of Scraping and Proxies

dnasedkina — Fri, 17 Mar 2023 13:27:10 +0000

Web scraping has become an essential tool for businesses that need to extract data from websites to gain insights into their target audience, competitors, and industry trends. Why is it important to use residential proxies to ensure data safety and compliance and where the future of residential proxies moving to?
In this episode of Ethical Data, Explained podcast, host Henry Ng is joined by Neil Emeigh, the founder and CEO of Sprious. They will broach trends and the future of the proxy and ethical web scraping industry; cases that make scraping look bad, and touch on the subject why it is important to be vocal about the positive applications of web scraping and maintain ethical data standards as a web scraping company.

Navigating the Grey Area

Regarding the reputation of scraping and proxies, opinions may vary depending on the individual. Scraping, particularly of publicly available information like social media, has faced scrutiny after the Cambridge Analytica case. Proxies, on the other hand, have an even worse reputation due to their use in illegal and unethical activities. While the industry as a whole is working to improve ethical standards, it is a challenging task for proxies.

“We refuse to use methods like botnet malware or taking advantage of vulnerable devices, unlike some providers who have recently been shut down by the FBI. We believe that our commitment to ethical practices sets us apart from others in the market who make similar claims but have been found to fall short in research papers. We acknowledge that our dedication to ethical business practices is a key factor in our success and are proud to stand by it” - Neil Emeigh

The legality of scraping has been tested in court cases like HiQ and 911.re. The future of scraping seems uncertain as social media companies face government pressure to protect personal data. However, for non-personal data like e-commerce, scraping is unlikely to face legal repercussions. Interestingly, large companies like Amazon, Target, and Walmart are paying proxy companies to help them scrape Facebook. As a result, there is a grey area where everyone tolerates scraping but uses anti-scraping technology to protect their data from competitors.

Exploring New Opportunities in the Proxy World

Other than the SEO and the various practices associated with it, the market is evolving to include other strategies such as social media account management and this could lead to further developments in the proxy market, with residential proxies playing a key role in enabling businesses to manage their social media accounts and perform other tasks requiring online anonymity. However, it is still to be seen how the market will evolve and what new use cases may emerge for residential proxies in the future. It has been suggested that scraping will remain the dominant force driving the proxy market, with the ability to collect vast amounts of publicly available data through AI technology being a prime example. This requires a significant number of residential proxies to be utilized, as the scale of data collection can be immense. While scraping will continue to be a key application, its uses will likely expand as the field of AI continues to grow.

Trends and Insights for Proxy Providers to Stay Competitive in the Next Few Years

In order for proxy providers to stay competitive in the next few years, they will need to stay ahead of the increasing anti-scraping technology put in place by large companies like Amazon. As the industry becomes more saturated, varying degrees of quality can be expected to enter the market. While public scraping may not be made illegal, scraping personal data may be in the future. To stay ahead of these challenges, proxy providers will need to become more hands-on and offer more guidance and support to end-users in terms of anti-fingerprint technology, user agents, rotating IPs, and good intervals.

Exploring the Competitive Landscape and Finding Opportunities for Success

When reselling proxies, the success rate is low due to the growing saturation of the market, with big recognizable brands dominating most of it. When navigating the competitive landscape of the proxy market, it is advisable for proxy providers to focus on finding a close-knit niche of customers who require their expertise and services. Rather than solely relying on website sales, providers should aim to help their customers succeed in their industries. Differentiation is also key to standing out from big brands with the same offerings and pricing tiers. It can be achieved by adding extra services or building tools that cater to specific industries. This can help providers stand out and succeed in a saturated market.

That’s it for today. To find out more, tune in to the full of this week’s episode! You can find links to Apple podcasts, Google podcasts, Spotify, YouTube on the official podcast webpage - Ethical Data, Explained!

Exploring the Ethics of Proxy Use: Creating a Safe and Trustworthy Data Environment

dnasedkina — Thu, 16 Mar 2023 14:36:25 +0000

Data safety starts with awareness. Compliance is not enough. Now more than ever, it is important to be both legal and ethical when working with data – especially when providing access to publicly available data. As questions around the legality and ethics of data collection abound, it’s crucial we look at how tools like proxies are used – and not just at what they are technically able to do.

In this week’s episode of Ethical Data, Explained Henry Ng, Dmitry Shkolnikov, and Alex Smirnoff explores what it means to be a trustworthy ethical data provider.

To Proxy or Not to Proxy: An Introduction to the Controversial World of Proxies

In the realm of ethical data, the accessibility to information is increasing rapidly. Although compliance with regulations is a fundamental starting point and a foundation for safety, it is insufficient to rely on a single party to ensure safety in data aggregation and collection. All parties involved must assume responsibility for maintaining safety and secure use of data. Educating individuals on the safe sourcing of publicly available data is just as important as ensuring that the data is legal and ethical. Proxy providers such as SOAX are contributing to the development of clear safeguards for both data providers and sources by implementing an educational system and checks.

“Proxies are often associated with malware and other equally harmful things but I think it's a wrong approach. Computers are also associated with malware but no one blames them in anything bad. Proxies are simply a tool that helps people from different countries access and collect data for the public good. It's not about anonymity or anything negative”.
Dmitry Shkolnikov

Proxies are not about anonymity, but rather a tool to help people access and collect data for public good. They aim not become a tool for bad actors and is proactive about security and data privacy for customers. At the same time proxy provider’s aim is not to collect user data rather than prioritize keeping both customers and partners safe. This is achieved through technical limitations and the use of great servers.

Data Safety: The Double-Edged Sword of Protecting User Privacy and Safeguarding Data Sources

The notion of data privacy and raising awareness about safeguarding it could be perceived as a two-sided issue. On one hand, individuals aim to utilize tools and gather data, while on the other hand, data is being collected and sourced from individuals and that can be used for both good and bad purposes. From a technical standpoint, it's difficult to determine how the data will be used. Therefore, a thorough investigation of the customer's business model and intentions is necessary to identify whether they can use proxy services or not. If a company intends to misuse the data or engage in malicious activities, access is denied. On the other hand, if they're a legitimate business creating products that will benefit the community, they're granted access. An example of such a scenario is price monitoring, where the data packages are encrypted, and the data content is unknown. It's impossible to tell if the data will be used for market intelligence or to bruteforce merchant accounts of marketplaces. In such cases, the compliance team requests more information about the company and scrutinizes their track record to determine if they're trustworthy before granting access to proxy services.

Ensuring Transparency and Ethics in Data Utilization and Proxy Tools: Best Practices for Providers, Players, and Resellers in the Data and Proxy Markets

In terms of improving transparency and security, it is important for providers to have a database of bad actors and implement KYC checks based on ethical principles. Transparency should be maintained for both players and providers in the market, and end suppliers of proxy services should be fully informed and involved in the partnership. Resellers also should implement the same KYC policies and regulations as the original proxy and not have lighter regulations. It is important for proxies to have open conversations with reseller partners and provide suggestions to ensure transparency on all sides and on each step of the supply chain.

It is essential to ensure that everyone involved is aware of the data collection process and that proper measures are in place to protect them. The use of proxy services should always be done with the individual's consent, as there are providers that use botnets instead. It is crucial to maintain ethical standards, check partners to ensure that the practices are ethical, and take the lead in raising awareness with their direct customers and promoting the idea of ethical use cases. Ultimately, the aim is to create a safer way to utilize data and improve the use of proxies by promoting awareness and education.
That’s it for today’s episode of our podcast. To find out more, tune in to the full of this week’s episode! You can find links to Apple podcasts, Google podcasts, Spotify, YouTube on the official podcast webpage - Ethical Data, Explained

If you do have any questions please feel free to reach out to us and if you have any questions around proxies or data we are happy to answer any of those thank you very much. See you next time!

Shifting the Status Quo of Cybersecurity

dnasedkina — Mon, 20 Feb 2023 09:32:14 +0000

Cybersecurity is possibly one of, if not the most important aspects of our internet usage and browsing. It is simultaneously both something we are all more than aware of, and yet somehow we all take it for granted. But how is it different for each and every individual? Does the world of cybersecurity favour particular types of people? And how much does your gender and culture play a role in that?

On the latest episode of our podcast, Ethical Data, Explained, host Henry Ng is joined by Jane Frankland, Founder of the Source Platform (for Women in Cyber). They discuss a multitude of topics, ranging from the sheer creativity of technology and AI; how cybersecurity has changed and evolved over the years, as well as what needs improving; and why diversity in the cyber space matters. Jane also discusses her love for statistical, analytic data, delving into how it enabled her to conduct such groundbreaking research into women in cybersecurity.

Why Diversity in Cybersecurity Matters

Jane reveals to us that she first created The Source Platform (for Women in Cyber) after noticing a lack of women and diversity in the cyber industry. “When I came into cybersecurity all those years ago, you really didn’t find women in the industry at all.” she divulges. “I remember meeting a female client once and thinking ‘wow’. I was just really shocked at the numbers, and then I could see them plateauing and I thought, ‘that’s really not a good sign.”

It was from this interaction that a fuse lit up in her head for her book, InSecurity, and after spending time at Cyber events, unable to find many women (if any) also attending them, she decided to take matters into her own hands. She used data, statistics, and analytics to really find out why it was that women weren’t attending or speaking at them.

But why does diversity within the tech space matter? How can it impact the industry in a beneficial way? Jane answers this question simply and matter-of-factly, stating that having more women in the workplace matters because it is far more natural than having only men; they think and approach risks differently, making them a valuable asset.

It’s not just women, either. Depending on your cultural background, you will almost definitely have a completely different perspective on life and risk-taking within your industry. Adding more people from all different backgrounds is the best way to grow your business and shift the status quo.

Some of the Biggest Changes in Cybersecurity

If you’ve been on the internet at any point over the last few years (which you almost certainly have), then you’ll most likely have seen an abundance of advertisements for all different kinds of VPNs (Virtual Private Network). Their very existence proves that society is becoming more and more weary as to their private data and just how easily someone could steal it. With so much going on online, it’s no wonder we’re becoming more protective of our data.

Jane acknowledges this, as there is now so much more risk involved with just browsing the internet. You never know what you’re going to come across and who’s lurking in the background, waiting to take your data. But, it’s important to remember, it’s not all looming doom and paranoid gloom, as it’s the people like Jane who are working hard to keep you safe online.

Jane says that another one of the biggest changes in cybersecurity is the language surrounding it. What was once known as ‘network security’, then ‘internet security’, is now known as ‘cybersecurity’. With so much growth and evolution within the cyber world, it’s no wonder the language had to adapt and fit these changes.

The Sheer Creativity of AI

Speaking of evolution and change, a massive one is the sudden growth of AI technologies (ChatGPT springs to mind straight away), something which Jane describes as creative. “Tech, to me, is so creative,” she says, her excitement for the topic betraying her. “AI is so interesting, particularly over the last few months with ChatGPT - it’s so powerful. It can really add and contribute.”

However, she also acknowledges why some may be hesitant towards AI’s meteoric rise, but why ultimately, we need to accept and flow with it: “We don’t like change, and that’s a human trait. Humans work really hard to maintain the status quo; change means risks and it opens up the danger. We need to move with technology as opposed to being more obstructive with it. We need to be breaking down the doors.”

To find out more, tune in to the full episode of this week’s episode of Ethical Data, Explained:
Apple Podcasts
Spotify
Google podcasts
YouTube
For more info reg the podcast, episodes and guests - welcome to the official podcast webpage.

CTA: if you know someone whose knowledge, experience and expertise could make them an interesting guest to discuss data and data collection - let me know in the comments! I am happy to reach out and invite them for a talk🙂

The Legalities of Data Collection - are you getting fined?

dnasedkina — Tue, 07 Feb 2023 08:13:26 +0000

On this episode of Ethical Data, Explained, Peter Craddock, Partner of Keller and Heckman, joins the discussion of personal data and the consequences of personal data infringement.

Peter previously worked as a lawyer before transitioning to software development, and presently works at Keller and Heckman as a Legal Technologist. He has continually expressed how much he enjoys helping clients achieve their digital ambitions by building legal and technical solutions. His mission is to make Keller and Heckman a top company in the sector and help their clients use data more efficiently and safely.

Episode covers:

1. Protecting the Personal Data of the Everyday Person

Privacy has always been a point of contention among the public, not just in the digital world. Even before Edward Snowden’s infamous leaks, there was an abundance of scepticism regarding the information we share with the online world, so, when it comes to the legalities of data collection, we must first acknowledge that everyone’s personal data is to be kept private and secure so that it doesn’t fall into the wrong hands. Even Steve Jobs himself hammered home the importance of privacy, stating, ‘Let them know precisely what you’re going to do with their data.’

Peter Craddock is an avid advocate for warning the public about scams and phishing emails; everyone needs to be aware of these things. Oftentimes, these phishing scams will get a hold of data leaks, or they might even buy data from third parties, resulting in (sometimes) very sophisticated, convincing scams that even the most eagle eyed internet user might be fooled by.

In fact, Peter even discusses a phishing scam that he was recently quite impressed by. ‘Sometimes, depending on my mood, either I try to play around and play along with them, and try to figure out how far they're willing to go in their explanation; or sometimes I'll just cut the call immediately, but this time I thought I'd play along for a bit,’ he said. ‘This was the most convincing phishing call I'd had in a while. They told me that I made an Amazon purchase for an iPhone, and it was an unusual transaction for me. So I said, “I'm sorry, but I'm not interested. Thank you.” They were trying to elicit a response of fear and susceptibility on my part. But I thought that the way that they responded to my questions was a lot smarter than what I'd seen [before]. I thought it was noteworthy.’

Were it not for Peter’s knowledge as a Partner of Keller and Heckman, he may have fallen for it, which is even more reason we should protect the data of those uneducated in the subject. He said, ‘It seemed like a very big coincidence that was happening just after a massive leak had been suggested. So I wondered perhaps it's related to that.’

2. How Can We Ensure Our Data is Protected?

Despite GDPR and the growing awareness of how important our private data is, we still need to make sure we are doing our utmost to protect it. But what actions can we take to ensure this protection? Ideally, it’s the organisations in charge of our data who should be taking those extra steps, and although they have put effort into it, data leaks still happen, making everyone with a phone number or an email vulnerable.

According to Peter, one way we can do this is by making use of AI aids to monitor and prevent any sort of data infringement as a result of these leakages. Organisations must also come together to ensure they are doing everything in their power to protect people’s data; they can work together to formulate security measures in this respect.

But, he does explain that there is an ongoing debate as to what exactly personal data is: ‘I do have a few ongoing cases where we are addressing the issue of what personal data is. Data protection legislation has been around for basically 30 years now, you'd think that some of these basic concepts have been sufficiently examined, but it's not yet the case. We still do not have sufficient clarity. So we still do have a tension. Some authorities are interpreting it in a very broad manner, some in a more restrictive manner. It's a fascinating topic, I think, because there are lots of situations.’

3. Keller and Heckman’s DeFine Tool

The DeFine Tool is Keller and Heckman’s program that allows them to calculate the fine penalties in cases of data infringement. This is an incredibly useful, resourceful tool, allowing both the firm and the client to know and understand the extent of the offence committed.

When discussing the tool, he said, ‘I've been building tools to help myself and clients for some time. I love it if a client says, “I've got an issue and can you help me?” And then, if I find out that it's something that they can do on their own, if I give them a tool, if it's a more rudimentary question, or if it's a specific process where my added value is more limited, then my added value is building the tool for them. Then they've got something, and you empower the clients. They can do the calculation or the assessment on their own, and that, I think, is a great way to help them.’

Listen to this episode of Ethical Data, Explained to learn more about how web scraping companies can maintain ethical standards and for insights on potential changes that could transform the industry.

You can find "Ethical Data, Explained" on

Apple Podcasts
Spotify
Google podcasts
YouTube

For more info reg the podcast, episodes and guests - welcome to the official podcast webpage.

Responsible Web Scraping: Challenges and Approaches

dnasedkina — Mon, 06 Feb 2023 09:03:11 +0000

“Web scraping is neither legal nor illegal. It's how you use it and what you scrape.” - Ondra Urban, COO @ Apify

If you’re new to the web scraping field or have been hurt in some way by web scraping, it’s difficult to see it in a positive light. Trust me; you’re not alone. Back in 2018, when web scraping and data collection emerged, it felt illegal in nature. However, with the recent rise in web scraping companies deploying ethical standards, the nature of web scraping has mostly morphed into a force for good.

On the inaugural episode of the podcast Ethical Data, Explained, Ondra Urban (COO at Apify but preferring the title “Chief Debugging Officer”) discusses how web scraping companies can maintain ethical standards. He explains how debugging and resolving problems can create a more accessible and programmable web. As a bonus, Ondra gives his thoughts on the implication of the HiQ vs. LinkedIn case on the web scraping industry and possible innovations in the field.

Insights on ethical data collection for businesses:

Web scraping is merely a tool; it’s you, as a company, that decides what to do with it. Therefore it could be a force for both good and evil, depending on who wields the tool.
No US federal laws currently ban web scraping, and while this does not imply that the US government has officially declared web scraping legal, it is a step in the right direction. However, it also means that the best way that web scraping companies can thrive in the field is to do their data harvesting ethically.
“Safe harbor” is important for a web scraping company. Safe habor protects you from getting hit by the consequences of your users' actions. For instance, if a user logs onto your web scraping platform and does something illegal, safe harbor protects you from that action – unless you were aware of it beforehand.
Look to provide value, and you’ll build trust and a positive image for your brand.

Possible Innovations in the Web Scraping Industry:

Speaking on the HiQ vs. LinkedIn case, Ondra highlights that while it affirmed that scraping public data is not a criminal offense, it still doesn’t entirely validate web scraping. Nonetheless, it is a good start that may spark up and shape the future of the web scraping space.
AI and web scraping: Ondra posits that AI won’t replace programmers in the web spacing field. While AI brings an innovative touch to any field it enters, it won’t make significant transformations in web scraping: "AI can take you 80%, 85% there, except for specific use cases," says Ondra. In other words, depending on the context, you might find AI-collected data useful a lot of the time. Unfortunately, not every piece of information gathered by AI ends up being useful.

You can find "Ethical Data, Explained" on

Apple Podcasts
Spotify
Google podcasts
YouTube

For more info reg the podcast, episodes and guests - welcome to the official podcast webpage.

Ethical Data, Explained

dnasedkina — Tue, 31 Jan 2023 10:14:55 +0000

Hello Dev.to community,

Let me introduce you to a brand new podcast "Ethical Data, Explained" dedicated to data, web scraping and data collection in all their forms and shapes!

In this podcast, we aim to debunk common myths surrounding web scraping and answer questions like: Is web scraping legal? Is it ethical to use proxies for data collection? Why do companies scrape data? etc.

We invite tech entrepreneurs, data security experts, lawmakers, and hackers to look at data and public data collection from various points of view to unlock safe and ethical web scraping for businesses.

Join us as we chat with experts about data, web scraping, proxies, ethics, and security.

You can find "Ethical Data, Explained" on

Apple Podcasts
Spotify
Google podcasts
YouTube

For more info reg the podcast, episodes and guests - welcome to the official podcast webpage.

What is an Anonymous Proxy and VPN?

dnasedkina — Fri, 27 Jan 2023 10:28:58 +0000

The primary function of a proxy is to make users anonymous and secure their privacy. But in reality a proxy can reveal the true IP address or inform a website that its visitor uses a proxy.
So what is the deal with an “anonymous” proxy?

What is an Anonymous Proxy?

A proxy server is used when a user needs to access network resources, such as the Internet. A user connects through a proxy server instead of directly connecting to the web hiding the true identity.
Some proxy types (transparent proxies) allow websites to see the real IP address associated with each host request, non-transparent proxies (anonymous) do not show the original IP address of a web server's visitor but reveal that a user uses a proxy to reach the web source.

An anonymous proxy, however, can do more. Use cases are widely known:

an anonymous proxy keeps private the browsing history;
helps to avoid content blocks and targeted marketing, and it blocks online ads;
provides access to geo-restricted websites;
runs automated tasks;
it speeds up loading times when caching configuration is set up.

How Does Anonymous Proxy Work?

An anonymous proxy like other proxy types serves as a middleman between a user and a web server. It intercepts browsing requests and routes them through a proxy server returning results to a user. The information a user sends or receives does not go from the original IP address but rather from a different server. The user’s data remains hidden, and the websites get them for somebody else. So an anonymous proxy allows surfing the Internet without revealing the real IP address.

What levels of proxy anonymity exist?

The term “anonymous “ is often used for marketing purposes thus the concept is misjudged by some users. Anonymous proxies are not completely anonymous thereby are not fully secure.

One must be careful by handling sensitive data using anonymous proxies, because their levels of anonymity differ and depend on their types.

Anonymous proxies can be divided into the three groups:

Transparent Proxy

Transparent proxies pass all information about a user and show a user’s true IP address. These are often used in schools that block access to websites with disturbing content. Companies use transparent proxies to route traffic and control which websites their employees can address while on work. Transparent proxies are often used in public wifi networks to authenticate users. When connecting to the Internet via public wifi users are redirected to a sign-in page and do not even know that they access the web through a proxy.
The connection through a transparent proxy is not encrypted and thus is not secured.

Anonymous Proxy

An anonymous proxy does not reveal users’ true IPs and their locations. However they inform websites that their visitors use a proxy, thus they can be easily blocked by a website that limits proxy access.
If you would like to access local content in other locations you need to use an elite proxy.

Elite Proxy

Elite proxies give a user the highest level of anonymity. They do not pass any information about a user: neither their true IP address nor personal data and they do not identify themselves as a proxy while making network requests.
Why use high-anonymous proxies when there are anonymous proxies? As many websites block anonymous proxies by default identifying them as proxies, using elite proxies allows one to avoid the proxy check and access any source a user sends a request to.

NB:
As for sensitive data security, elite proxies are the same as anonymous proxies. Both belong to a third party. If it is not a trusted seller a user might end up being a victim of a cybercrime.

What’s the Difference between Anonymous Proxy and VPN?

Both — a VPN anonymizer and an anonymous proxy — hide the source of browsing activity by routing traffic to the server that conceals the real IP address and accesses the target website as a middleman.
Contrary to what is believed a VPN service does not provide a user ultimate anonymity. VPN acts as a tunnel: once addressing the Internet through VPN an encrypted connection occurs between a VPN server in the chosen location and a user’s network device.

An anonymous proxy does not guarantee full anonymity while browsing the Internet as well as VPN. Their main task is to encrypt the exchanged data between a device and the network, so a user’s data remains unusable to hackers.

The main difference between VPN and an anonymous proxy is that VPN secures traffic from a single IP address, while a proxy can utilize millions of IPs for various automated scraping and research tasks.

How to Set Up and Use Anonymous Proxy?

An anonymous proxy can be set up in a browser (Explorer, Mozilla, Chrome, Safari), on Mac and Windows operating systems, and on IOS and Android devices. The manual configuration is seamless and effortless.

Windows
Go to “Windows Settings” Menu → “Settings” → “Network & Internet” → “Proxy”. On the manual setup section set the “Use a proxy server” → “On” option → Enter the address of your proxy server and a proxy port number → Click “Ok”.

Mac
Go to “System Preferences” → “Network” → “Advanced” → “Proxies” → Select a protocol to configure → click “Secure Web Proxy” (HTTPS) → Type the address of the proxy server and its port number → If the proxy is password-protected, enter the account name and password → Click "Ok".

Chrome
Chrome uses Windows 10/11 or macOS proxy settings by default. If you wish to change your Chrome proxy settings, follow the following steps:

Go to “Settings” on the Chrome toolbar → “Advanced” → “Settings” → “System” → Open your computer's proxy settings → “Manual proxy setup” → Enter the address of your proxy server and a proxy port number → Click “Save”.

Safari
Just like in Google Chrome Safari operating system uses proxy settings by default. If you would like to change a proxy configuration, follow these steps:

Go to the Safari toolbar → “Preferences” → “Advanced” tab → “Proxies” → “Change settings” → Go back to the instructions on macOS on setting up a proxy server and follow them.

Firefox
Firefox does not use proxy settings by default from the OS. Here you can know about Firefox Proxy setup:
Open the “Firefox menu” → “Internet options” → “Advanced” → open the Network tab → Go to the “Connection settings” → “Manual proxy configuration” → Enter the address of your proxy server and a proxy port number → Click "Ok".

Edge
Edge is a built-in browser in Windows 10/11, and the browser will use the settings in Windows 10/11 by default unless you change them.

Go to “Settings” → “Advanced settings” → “Open proxy settings” → Go back to the Windows 10/11 instructions and follow them to complete the configuration.

Android
It is better to use a VPN service on Android because when using a proxy server you need to configure the settings for each network you use. VPN thereby will cover all traffic coming through a mobile device.

Open the “Android settings” → the long press on the name of the “Wi-Fi network” → “Modify Network” → Proceed to "Advanced Options” → Click on "Manual" to change the proxy server settings and then enter your hostname and proxy port → Click “Save”.

IOS
Go to "Settings" → “Wi-Fi” → Select the active Wi-Fi network → Go to the “HTTP proxy” → Click on "Configure Proxy” → Tap on the manual → Enter the hostname and proxy port → If needed enter the authentication username and a password.

There are a few reasons why to remain anonymous while surfing the Internet.

Anonymous proxies help to avoid targeted ads and aggressive marketing campaigns. They also allow one to bypass geo-blocking and access restricted content.

Enterprises deploy proxies to filter all incoming and outgoing requests, boost security, prevent employees data theft, manage site access, ensure load balance, and cache files to make network devices run faster and more efficiently.

Anonymous proxies also run automated tasks: they help to avoid captcha, and bypass firewall restrictions. With anonymous proxies it is easier to manage scrape tasks on different levels, compare prices, manage multiple accounts, monitor SEO, test ads, enhance bot activity and more.

Each request a user sends to the network via HTTP protocol sends a header. It records the information about the device. A proxy alters the header’ data at different levels based on its anonymity level.

There are several types of proxies, including transparent proxies, that allow websites to understand what visitors view their sites and see their IPs; non-transparent (anonymous) proxies that do not show the original IP address but manifest themselves as proxies and so might be blocked by particular websites; and the safest — elite proxies.

Despite the above-mentioned, anonymous proxies are irreplaceable while scraping the web, introducing automated business tasks and other online activities that require the usage of IP pools.

This post was originally published on SOAX blog.

Proxy for Instagram: What is It and How to Get It?

dnasedkina — Tue, 24 Jan 2023 08:13:33 +0000

Instagram as one of the internet’s biggest traffic sources is the most popular and favorite social media platform. Here you can present your products, find new loyal followers, and build a solid online presence. No wonder retailers and brands consider Instagram as a perfect opportunity to raise their profits.

What is a Proxy for Instagram?

A proxy server is a remote device that changes your IP address. Being connected to a proxy you reroute traffic coming from multiple devices to this proxy. The destination server — a website or a social media platform — sees only a proxy’s IP you are connected to, whilst your true IP address is hidden. Hence you avoid getting blocked and going around antibot policies and anti-spam detection systems.

An Instagram proxy has the same purposes. It hides your location, secures your private data, and grants access to the target server bypassing its restrictions. The true value of an Instagram proxy is concealed in its name. It is optimized to be used only to work specifically on the social media platform.

What Proxy for Instagram Can Be Used for?

There are several reasons to use a proxy for Instagram, but first and foremost a proxy ensures social media accounts safety and longevity.

NB: Each Instagram account needs its own residential proxy. Otherwise, when using more than one account from the same proxy, Instagram algorithms will consider these actions as spam. If you’re using mobile proxies, you can use 5-10 accounts per proxy.

So what are the benefits of using Instagram proxies?

Instagram unblocking
Instagram proxies solve the issue of GEO blocking. If a country or a city blocks your access to the social media platform, a proxy is a solution. VPN can help you here as well yet still, there is a difference between a VPN and a proxy, for example, a proxy is invaluable when collecting data of any scale.

By using Instagram proxies to bypass geo-restrictions you can target the audience on a global scale and hit a particular geo-location when needed making a more favorable business offer.

Instagram scraping
With more than 1,3 million active users per month, IG is a high-profile source of valuable information. You can scrape Instagram hashtags, photos, and profiles to build up effective marketing strategies.

Multiple Instagram account management
If your business needs multiple accounts, Instagram proxies will let them grow by gaining followers’ likes and leaving a thousand comments (marketing automation tools will help you to achieve it).
You can engage an audience on a much larger scale, target it from different locations, and send automated texts, ads, and promotional content from hundreds of accounts at once, thus speeding up the marketing process and bypassing strict Instagram policies. Thus deploying a proxy in AdsPower would lead to secured and successful multi-accounting from a single IP.

Building up followers
The growing number of followers is the result of successful social media account management. By building up a brand audience you grow followers base countable in hundreds of thousands.

Which Type of Proxy Works Best for Instagram?

The Instagram algorithms are designed to stop curious brands from harvesting valuable data about their customers. To get around the Instagram restrictions brands use proxy services to mask their true location and identity. There are several types of proxies to scrape Instagram, the problem is choosing the right type that is worth the money spent.

Datacenter (dedicated) proxies
Are hardly being used to working on Instagram these days, because they are not attached to real devices and are not assigned by an ISP, thus Instagram algorithms will ban you and the whole IP range. The only reason to work with data center proxies on Instagram is for scraping, but they are not good neither for Instagram account management nor for Instagram account growth.

Residential proxies
Are real devices with their IPs coming from ISP. Residential proxies both rotating and static ones are suited to work on Instagram. Rotating proxies are better for Instagram account creation whereas static IPs work well for account management.

Mobile proxies
Are the most trustable ones. They belong to real mobile devices and are rarely getting blocked. Mobile proxies, however, are expensive and if you are running hundreds of Instagram accounts it will cost a bundle.

How to Get a Proxy for Instagram Bots?

Proxies for IG are the residential proxies that are optimized to work with the social media platform. When you need to scrape IG or manage multiple IG accounts purchase a suitable proxy plan and put the purchased proxies into a bot. It will do the rest.

What are The Best Instagram Proxy Providers?

Mobile and residential proxies with sticky sessions for mass IG accounts:

Smartproxy
Rotating residential IPs for multi-accounting.
Large rotating residential proxy pool, 195+ countries, no subnets, an easy setup process. The IPs are from both desktop and mobile devices. Different tools for IG automation and affordable price plans ($75/5 GB ($15/GB). However the residential IPs are still P2P, so you have sticky sessions up to 30 min and then proxies rotate.

Brightdata
Mobile and static residential proxies with multiple features.
The largest proxy provider. Country and city targeting, ASN, and carrier targeting. The best for account creation, management, and scraping. Static sessions are available. No monthly plans: just pay as you go. Proxies however are costly ($20/GB + $0.5/IP for static residential proxies). The useful extra features (like ASN targeting) add to the price.

GeoSurf
Static and rotating proxies, thousand of cities.
Great speed. The IPs pool is small but well maintained. P2P residential proxies. Great and stable performance. Not perfect (better use static option for that) but suitable for account management with sticky sessions that last up to 30 min. Precise city targeting. One of the most expensive proxy providers ($450 for 38 GB ($11.85/GB)), but some unique features (such as unlimited bandwidth) are at no extra cost.

SOAX
Mobile and residential proxies with highly adjustable rotation.
180+ countries with highly precise targeting options. An average price ($75 for 5GB ($15/Gb). IP whitelisting is required. Some useful features come as paid add-ons. ASN targeting is at no extra cost. Plenty of options for proxy rotation: measured in seconds, minutes, and as long as the IP is available. An option is to keep the same IP address after it loses connection. Suitable for account creation and management.

The Social Proxy
High-quality 4G mobile private proxies.
Legitimate mobile proxies from real users, high level of rotation. Developed specifically for social media automation. Comes with a REST API that makes extra features and automatic rotation possible, and integrates well with most of the powerful social media automation tools to improve IG automation. Locations are in Germany, Austria, America, the UK, and Texas (US). Unlimited bandwidth, simple pricing model (€90/proxy/a month), free trial.

Dedicated data center proxies for growing Instagram accounts:

Zyte’s Smart Proxy Manager
A rotating proxy API for Instagram scraping
Datacenter IPs, a proxy manager with integrated throttling, proxy rotation, automatic retries, header, and cookie management — an optimal choice for IG scraping. Paying for successful requests only ($99 for 200k successful requests). The cheaper plans with fewer features are available.

High proxies
Datacenter proxies for Instagram scraping.
Proxies are located in the US, CA, UK, NL, Germany, France, and Italy. The customer support is responsive and fast. Private HTTP/HTTPS proxies are fast and stable. Support both user/password and IP-Auth. Fast connection speed and a hassle-free setup. Price is higher as compared to the competitors ($3,20/social media proxy/a month).

The primary purpose of Instagram proxies is to avoid blocking by remaining unrecognizable for the social media platform, whether you use it to monitor a brand, generate leads, raise sales, or perform marketing campaigns.
Instagram proxies help to manage multiple accounts to interact with the audience no matter where in the world it is located. Using automation bots you can easily increase your traffic and benefit from likes, sharings, and comments.
Being able to create and manage multiple Instagram accounts has major benefits for businesses, whether you are a marketer manager or a business owner promoting your products and services across the globe.

This post was originally published on SOAX blog.