Posted on
Sometimes in ecommerce web scraping you scrape the same product but with different names like :
Msi RTX 2070 Ti
Nvidia rtx 2070ti MSI 8gb
And more complicated examples, I wonder if there is any neural net to classify these type of products
For further actions, you may consider blocking this person and/or reporting abuse
Thanks for visiting DEV, we’ve worked really hard to cultivate this great community and would love to have you join us. If you’d like to create an account, you can sign up here.
Lukka -
Jasper Samuel J -
Julien Simon -
Crazy Codigo -
Once suspended, sihamza will not be able to comment or publish posts until their suspension is removed.
Once unsuspended, sihamza will be able to comment and publish posts again.
Once unpublished, all posts by sihamza will become hidden and only accessible to themselves.
If sihamza is not suspended, they can still re-publish their posts from their dashboard.
Once unpublished, this post will become invisible to the public and only accessible to Jouini Hamza.
They can still re-publish the post if they are not suspended.
Thanks for keeping DEV Community 👩💻👨💻 safe. Here is what you can do to flag sihamza:
Unflagging sihamza will restore default visibility to their posts.
Top comments (4)
Rather than a neural network, I think you should look at more differentiating features from say the description and remove the duplicate products. For instance, the model number will be the same for the above laptops, if you are using a custom scraper, it should be doable, find the model number from the details page and make it the primary key in your db, that should do.
it's doable for 2 3 websites but i am scraping like 8 websites
Yes I think it's still doable, but I will still try to find any network if available. Also, you will be adding great overhead if you're using a neural net not to mention the r sources you'd need in case you need to run it on server
A good tokenizer and pairwise Levenshtein distance should be a good starting point. en.wikipedia.org/wiki/Levenshtein_...