The latest articles on DEV Community by hfrnssc (@hfrnssc). https://dev.to/hfrnssc https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F510184%2F651fdc92-da7b-4151-9f05-9e366f05413e.png https://dev.to/hfrnssc en hfrnssc Sun, 08 Nov 2020 04:20:23 +0000 https://dev.to/hfrnssc/how-can-i-solve-the-tokenization-process-with-r-48gb https://dev.to/hfrnssc/how-can-i-solve-the-tokenization-process-with-r-48gb <div class="ltag__stackexchange--container"> <div class="ltag__stackexchange--title-container"> <div class="ltag__stackexchange--title"> <h1> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--pTF_nE4a--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://practicaldev-herokuapp-com.freetls.fastly.net/assets/stackoverflow-logo-b42691ae545e4810b105ee957979a853a696085e67e43ee14c5699cf3e890fb4.svg" alt=""> <a href="https://stackoverflow.com/questions/64730524/how-can-i-solve-the-tokenization-process-with-r" rel="noopener noreferrer"> How can I solve the tokenization process with R? </a> </h1> <div class="ltag__stackexchange--post-metadata"> <span>Nov 7 '20</span> <span>Comments: 1</span> <span>Answers: 0</span> </div> </div> <a class="ltag__stackexchange--score-container" href="https://stackoverflow.com/questions/64730524/how-can-i-solve-the-tokenization-process-with-r" rel="noopener noreferrer"> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--5MiFESHx--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://practicaldev-herokuapp-com.freetls.fastly.net/assets/stackexchange-arrow-up-eff2e2849e67d156181d258e38802c0b57fa011f74164a7f97675ca3b6ab756b.svg" alt=""> <div class="ltag__stackexchange--score-number"> 0 </div> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--Rk_a5QFN--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://practicaldev-herokuapp-com.freetls.fastly.net/assets/stackexchange-arrow-down-4349fac0dd932d284fab7e4dd9846f19a3710558efde0d2dfd05897f3eeb9aba.svg" alt=""> </a> </div> <div class="ltag__stackexchange--body"> <p>I want to process the text from this</p> <p><a href="https://i.stack.imgur.com/uAFlZ.png" rel="nofollow noreferrer"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--mvgVlmUD--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://i.stack.imgur.com/uAFlZ.png" alt="enter image description here"></a></p> <p>to this with removal of regular expression, like emoticon symbol, numbers, punctuation, etc.</p> <p><a href="https://i.stack.imgur.com/BRk9c.png" rel="nofollow noreferrer"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--PhFpwWkX--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://i.stack.imgur.com/BRk9c.png" alt="enter image description here"></a></p> <p>I have tried with this code</p> <pre><code>library(dplyr) library(tidyr) library(tidytext) library(textdata) library(purrr) library(csv) sentanalysis &lt;- read.csv('crawling_commuterline.csv', header=TRUE, sep=";", encoding="UTF-8") sentanalysis tweetdt = sentanalysis %&gt;% select(tw) head(tweetdt) tidytext::unnest_tokens(read.csv("crawling_commuterline.csv", stringsAsFactors = FALSE),word,tw) </code></pre> <p>…</p> </div> <div class="ltag__stackexchange--btn--container"> <a href="https://stackoverflow.com/questions/64730524/how-can-i-solve-the-tokenization-process-with-r" rel="noopener noreferrer">Open Full Question</a> </div> </div>