The latest articles on DEV Community by Anishka (@anishka0107). https://dev.to/anishka0107 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F390825%2Fe899813f-9805-472b-b2cc-6c3fe57050ab.png https://dev.to/anishka0107 en Anishka Wed, 20 May 2020 15:21:24 +0000 https://dev.to/anishka0107/wunner-54nm https://dev.to/anishka0107/wunner-54nm <h2> Wunner </h2> <p>Wunner is a simple terminal-based web search engine I built to understand the basic working of a search engine.</p> <h2> Link to Code </h2> <div class="ltag-github-readme-tag"> <div class="readme-overview"> <h2> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--vWogaON8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://practicaldev-herokuapp-com.freetls.fastly.net/assets/github-logo-28d89282e0daa1e2496205e2f218a44c755b0dd6536bbadf5ed5a44a7ca54716.svg" alt="GitHub logo"> <a href="https://github.com/Anishka0107"> Anishka0107 </a> / <a href="https://github.com/Anishka0107/Wunner"> Wunner </a> </h2> <h3> A toy search engine </h3> </div> <div class="ltag-github-body"> <div id="readme" class="md"> <h1> Wunner</h1> <p>A toy search engine that searches the web inside your terminal :p</p> <h2> Features</h2> <ul> <li>Implemented in C++14.</li> <li>Crawls webpages progressively starting from seed URL(s).</li> <li>Parses the documents and the query, trying to generate more appropriate results.</li> <li>Builds an index (hash map) for the parsed documents.</li> <li>The crawled documents and index are refreshed periodically.</li> <li>Autocompletes query using a trie, based on most recently asked queries.</li> <li>Maintains two threads, to allow refreshing the index and querying simultaneuosly.</li> <li>Generates most relevant results in order ranked on the basis of harmonic mean of PageRank (to get the importance of webpage) and Okapi BM25 (to get query-based result) algorithm ranks.</li> <li>Provides query suggestions (only when the input query does not generate any results), on the basis of common incorrect and correct words. Ranks them using n-gram algorithm and edit-distance DP to compare two strings.</li> </ul> <h2> Steps to Run</h2> <p>Command to run : <code>wunner_search</code> (make sure your…</p> </div> </div> <div class="gh-btn-container"><a class="gh-btn" href="https://github.com/Anishka0107/Wunner">View on GitHub</a></div> </div> <h2> How I built it </h2> <p>It is purely written in C++14. It crawls webpages progressively starting from seed URL(s). Then it pre-processes the documents and builds an index for the parsed documents. The crawled documents and index are refreshed periodically. It autocompletes query using a trie, based on most recently asked queries. It generates the most relevant results in order ranked on the basis of harmonic mean of PageRank (to get the importance of a webpage) and Okapi BM25 (to get query-based results) algorithm ranks. It also provides query suggestions when the input query does not generate any results, on the basis of common incorrect and correct words (ranked using n-gram algorithm and edit-distance).</p> <h2> Additional Thoughts / Feelings / Stories </h2> <p>It was fun developing this project as it was built from scratch and allowed me to learn about different data structures and use them practically.</p> <p>Final Note: Stay healthy, stay safe!</p> octograd2020 devgrad2020