DEV Community

Crawling Websites in React-Native

K on October 16, 2017

Coming from years of web developing React-Native feels like a fresh start to me. You get better access to native functionality AND you have fewer r...
Collapse
 
kayis profile image
K

Glad this article is still helpful after all that time :D

Collapse
 
acaraccioli profile image
Acaraccioli

Hello K, great post learned a lot I didnt know this was possible using fetch. Just one quick question. How would you manage if you wanted to fetch some quick data in front end (react) but had to enter information in an input tag and maybe even click a button? I hope you can help me out a bit

Collapse
 
kayis profile image
K

Glad you liked it.

I'd use React hooks.

function MyComponent(props) {
  const [info, setInfo] = React.useState("");
  const [remoteData, setRemoteData] = React.useState(
    "No data fetched yet!"
  );

  async function load() {
    const response = await fetch(
      "http://example.com?info=" + info
    );
    const text = await response.text();
    setRemoteData(text);
  }

  return (
    <div>
      <input
        value={info}
        onChange={(e) => setInfo(e.target.value)}
      />
      <button onClick={load}>Fetch</button>
      <textarea>{remoteData}</textarea>
    </div>
  );
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
acaraccioli profile image
Acaraccioli

That is great I think that might works thanks a lot! I'm just trying to figure out this error:
Access to fetch at 'MyUrl' from origin 'localhost:8100' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's. I've found to fix it adding {mode:"no-cors"} in the fetch call but the object returns null. Do you know anything about this kind of error?

Collapse
 
hardi_dev profile image
Hardiansa

Hi K, this awesome post.
So already try and it's working fine,

But I have some problem,
I try scrape from streaming service website (anime).

But there are no video tag inside the website.

So I try to inspect element, then I saw that website need to click "play" button then I got ifarame embedded html document with video tag.

So how can i do click on cheerio then get embeded video?

Thanks

Collapse
 
kayis profile image
K

Sorry, I don't know if Cheerio works with JavaScript sites.

One way to solve this would be to check if you could calculate the video URL from the data that is already in the HTML.

Otherwise I don't know.

Collapse
 
bagustyo profile image
Bagustyo

why Async ? what if just fetch ?

Collapse
 
kayis profile image
K

You can use fetch without async/await. React-Native supports async/await, that's why I used it, but it isn't needed, you can use promises directly :)

Collapse
 
binaryforgeltd profile image
Bart Karalus

Nice one! I had no clue there was a jquery-like tool for RN. Very useful.

Collapse
 
crawlbase profile image
Crawlbase

Thankyou! Impressive breakdown of web crawling in React-Native! Your methodical approach and clear explanations make it accessible for anyone diving into this field. Don't forget to streamline your efforts with Crawlbase for enhanced efficiency.

Collapse
 
kwangmart profile image
Martin Sone

Hi K, could the above crawling applies to reactjs or only to react native?

Collapse
 
kayis profile image
K

Only React-Native, because you can't access other websites from within a browsers, just sites from the same domain or such that are CORS enabled.

Collapse
 
yunusist profile image
ynstl

No, it's not working. ERROR.
ESLint Parsing error: Unexpected token

Collapse
 
prakort profile image
Prakort Lean

Please go in depth, i couldn't get cheerio-without-node-native to work

Collapse
 
sixman9 profile image
Richard Joseph

There's also react-native-cheerio, I've not yet used it myself but, obviously, I'm doing the research, also.

Collapse
 
lilrajax profile image
lilraja-x

I'm new to this react native thing... I've tried your exact method as of now yet nothing is displayed on my react native mobile app.
I'm new to this so your help will matter alot.

Collapse
 
lilrajax profile image
lilraja-x

There's no error but also nothing's displayed.