The latest articles on DEV Community by Avnis Singh (@avnsihsingh). https://dev.to/avnsihsingh https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F805624%2Fa5cd02dc-ab10-4e2d-9754-b0c2c5964a28.jpg https://dev.to/avnsihsingh en Avnis Singh Tue, 23 Aug 2022 11:36:13 +0000 https://dev.to/avnsihsingh/spark-version-330-connect-with-kafka-3p30 https://dev.to/avnsihsingh/spark-version-330-connect-with-kafka-3p30 <div class="ltag__stackexchange--container"> <div class="ltag__stackexchange--title-container"> <div class="ltag__stackexchange--title"> <div class="ltag__stackexchange--header"> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--7Gn-iPj_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/stackoverflow-logo-b42691ae545e4810b105ee957979a853a696085e67e43ee14c5699cf3e890fb4.svg" alt=""> <a href="https://stackoverflow.com/questions/73457943/spark-version-3-3-0-connect-with-kafka" rel="noopener noreferrer"> Spark Version 3.3.0 connect with kafka </a> </div> <div class="ltag__stackexchange--post-metadata"> <span>Aug 23 '22</span> <span>Comments: 1</span> <span>Answers: 0</span> </div> </div> <a class="ltag__stackexchange--score-container" href="https://stackoverflow.com/questions/73457943/spark-version-3-3-0-connect-with-kafka" rel="noopener noreferrer"> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--Y9mJpuJP--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/stackexchange-arrow-up-eff2e2849e67d156181d258e38802c0b57fa011f74164a7f97675ca3b6ab756b.svg" alt=""> <div class="ltag__stackexchange--score-number"> 0 </div> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--wif5Zq3z--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/stackexchange-arrow-down-4349fac0dd932d284fab7e4dd9846f19a3710558efde0d2dfd05897f3eeb9aba.svg" alt=""> </a> </div> <div class="ltag__stackexchange--body"> <p>I am trying to connect to my Kafka from spark but getting an error Kafka Version: 2.4.1 Spark Version: 3.3.0</p> <p>I am using jupyter notebook to execute the pyspark code below:</p> <pre><code>from pyspark.sql.functions import * from pyspark.sql.types import * #import library import os from pyspark.sql import SparkSession os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages</code></pre>… </div> <div class="ltag__stackexchange--btn--container"> <a href="https://stackoverflow.com/questions/73457943/spark-version-3-3-0-connect-with-kafka" class="ltag__stackexchange--btn" rel="noopener noreferrer">Open Full Question</a> </div> </div> Avnis Singh Sat, 29 Jan 2022 08:04:49 +0000 https://dev.to/avnsihsingh/how-to-resolve-opensearch-unreachable-https1270019200-unable-to-find-valid-certification-path-to-requested-target-2hg1 https://dev.to/avnsihsingh/how-to-resolve-opensearch-unreachable-https1270019200-unable-to-find-valid-certification-path-to-requested-target-2hg1 <div class="ltag__stackexchange--container"> <div class="ltag__stackexchange--title-container"> <div class="ltag__stackexchange--title"> <h1> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--7Gn-iPj_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/stackoverflow-logo-b42691ae545e4810b105ee957979a853a696085e67e43ee14c5699cf3e890fb4.svg" alt=""> <a href="https://stackoverflow.com/questions/70882768/how-to-resolve-opensearch-unreachable-https-127-0-0-19200-unable-to-fin" rel="noopener noreferrer"> How to resolve "OpenSearch Unreachable: [https://127.0.0.1:9200/][ unable to find valid certification path to requested target? </a> </h1> <div class="ltag__stackexchange--post-metadata"> <span>Jan 27 '22</span> <span>Comments: 1</span> <span>Answers: 0</span> </div> </div> <a class="ltag__stackexchange--score-container" href="https://stackoverflow.com/questions/70882768/how-to-resolve-opensearch-unreachable-https-127-0-0-19200-unable-to-fin" rel="noopener noreferrer"> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--Y9mJpuJP--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/stackexchange-arrow-up-eff2e2849e67d156181d258e38802c0b57fa011f74164a7f97675ca3b6ab756b.svg" alt=""> <div class="ltag__stackexchange--score-number"> 0 </div> <img src="https://res.cloudinary.com/practicaldev/image/fetch/s--wif5Zq3z--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/stackexchange-arrow-down-4349fac0dd932d284fab7e4dd9846f19a3710558efde0d2dfd05897f3eeb9aba.svg" alt=""> </a> </div> <div class="ltag__stackexchange--body"> <p>Hi I am trying to ingest data from logstash(oss) to Opensearch but it seems I can't connect to Opensearch from logstash The error log:</p> <pre><code>[avs@localhost pipeline]$ ./bin/logstash -f config/pipeline/ipv4.conf -bash: ./bin/logstash: No such file or directory [avs@localhost pipeline]$ cd .. [avs@localhost config]$ cd .. [avs@localhost logstash-7.16.2]$ ./bin/logstash -f config/pipeline/ipv4.conf Using</code></pre>… </div> <div class="ltag__stackexchange--btn--container"> <a href="https://stackoverflow.com/questions/70882768/how-to-resolve-opensearch-unreachable-https-127-0-0-19200-unable-to-fin" rel="noopener noreferrer">Open Full Question</a> </div> </div> elasticsearch security linux opensearch