The latest articles on DEV Community by hiroyuki sato (@hiroysato). https://dev.to/hiroysato https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F49246%2Fcb9174bc-57f0-480d-b062-57ce9f6b757b.jpg https://dev.to/hiroysato en hiroyuki sato Mon, 18 Nov 2019 01:40:26 +0000 https://dev.to/hiroysato/digdag-workflow-engine-link-collection-13hf https://dev.to/hiroysato/digdag-workflow-engine-link-collection-13hf <h1> Digdag Workflow engine link collection. </h1> <h2> Official pages </h2> <ul> <li><a href="https://docs.digdag.io/getting_started.html">Digdag in 5 minutes</a></li> <li> <a href="https://www.digdag.io">Official site</a> SIMPLE, OPEN SOURCE, MULTI-CLOUD WORKFLOW ENGINE</li> <li><a href="http://docs.digdag.io">Documentation</a></li> <li><a href="https://github.com/treasure-data/digdag">GitHub</a></li> </ul> <h2> Third party operators. </h2> <ul> <li><a href="https://github.com/szyn/digdag-slack/commits/master">digdag-slack</a></li> <li><a href="https://github.com/retz/retz-digdag-plugin">retz-digdag-plugin</a></li> <li><a href="https://github.com/hiroyuki-sato/digdag-plugin-ssh">digdag-plugin-ssh</a></li> <li><a href="https://github.com/tamanyan/digdag-hangouts-chat">digdag-hangouts-chat</a></li> <li><a href="https://github.com/hiroyuki-sato/digdag-plugin-mysql">digdag-plugin-mysql</a></li> <li><a href="https://github.com/kimutansk/digdag-plugin-dfs_wait">digdag-plugin-dfs_wait</a></li> <li><a href="https://github.com/septeni-original/k8s_job">k8s_job</a></li> <li><a href="https://github.com/civitaspo/digdag-operator-emr_fleet">digdag-operator-emr_fleet</a></li> <li><a href="https://github.com/civitaspo/digdag-operator-athena">digdag-operator-athena</a></li> <li> <a href="https://github.com/takemikami/digdag-plugin-shresult">digdag-plugin-shresult</a> (<a href="https://translate.google.com/translate?sl=ja&amp;tl=en&amp;u=http%3A%2F%2Ftakemikami.com%2F2018%2F06%2F08%2FforeachDigdag.html">Blog(Use Google translate)</a>)</li> <li><a href="https://github.com/civitaspo/digdag-operator-cost_explorer">digdag-operator-cost_explorer</a></li> <li><a href="https://github.com/civitaspo/digdag-operator-param">digdag-operator-param</a></li> <li><a href="https://github.com/civitaspo/digdag-operator-ecs_task">digdag-operator-ecs_task</a></li> <li><a href="https://github.com/kencharos/digdag-plugin-azure">digdag-plugin-azure</a></li> <li><a href="https://github.com/kulmam92/digdag-plugin-mssql">digdag-plugin-mssql</a></li> <li><a href="https://github.com/civitaspo/digdag-operator-livy">digdag-operator-livy</a></li> <li><a href="https://github.com/yuhiwa/digdag-plugin-sshresult">digdag-plugin-sshresult</a></li> <li><a href="https://github.com/tosametal/digdag-plugin-s3-touch">digdag-plugin-s3-touch</a></li> <li><a href="https://github.com/NomadBlacky/digdag-plugin-datadog">digdag-plugin-datadog</a></li> <li><a href="https://github.com/supimen/digdag-plugin-glue">digdag-plugin-glue</a></li> <li> <a href="https://github.com/civitaspo/digdag-operator-pg_lock">digdag-operator-pg_lock</a> It enables control _parallel size, locks a specific resource and avoids duplicate execution. <a href="https://translate.google.com/translate?sl=ja&amp;tl=en&amp;u=https://tech.gunosy.io/entry/digdag-operator-pg_lock%23%E3%83%A6%E3%83%BC%E3%82%B9%E3%82%B1%E3%83%BC%E3%82%B9-Workflow%E3%81%AE%E9%87%8D%E8%A4%87%E5%AE%9F%E8%A1%8C%E3%82%92%E9%81%BF%E3%81%91%E3%81%9F%E3%81%84">Blog</a> </li> <li> <a href="https://github.com/katsuyan/digdag-plugin-myjdbc">digdag-plugin-myjdbc</a> Use custom JDBC(ex. PureData) library operator.</li> <li> <a href="https://github.com/st-tech/digdag-operator-gke">digdag-operator-gke</a> <code>gke&gt;</code> operator for the Google Kubernetes Engine.</li> <li> <a href="https://github.com/TrsNium/digdag-operator-store_gke_config">digdag-operator-store_gke_config</a> An operator for store a GKE configuration.</li> <li> <a href="https://github.com/DeNA/digdag-operator-bq-wait">digdag-operator-bq-wait</a> </li> <li><a href="https://github.com/supimen/digdag-plugin-glue">digdag-plugin-glue</a></li> <li> <a href="https://github.com/emanon-was/digdag-operator-aws-appconfig">digdag-operator-aws-appconfig</a> Digdag operator to store parameter from AWS AppConfig</li> <li> <a href="https://github.com/to-lz1/digdag-plugin-sentry">digdag-plugin-sentry</a> A Digdag plugin for sending errors to Sentry</li> <li> <a href="https://github.com/flinters/digdag_operator_snowflake">digdag_operator_snowflake</a> Digdag <code>snow&gt;</code> operator plugin to execute a query Snowflake.</li> </ul> <h2> Extensions </h2> <ul> <li> <a href="https://github.com/yoyama/digdag-basic-auth-mt">digdag-basic-auth-mt</a> Multi tenant basic-auth plugin for Digdag</li> </ul> <h2> Tools </h2> <ul> <li> <a href="https://github.com/TrsNium/digdag-worker-crd">digdag-worker-crd</a> A horizontal pod autoscaler for digdag worker</li> </ul> <h2> Third party digdag client </h2> <ul> <li> <a href="http://iroilog.net/2017/06/release-mog/">mog</a> Go</li> <li> <a href="https://github.com/saicologic/digdag_client">digdag_client</a> Ruby</li> <li> <a href="https://pypi.org/project/tdworkflow/">tdworkflow</a> Python</li> </ul> <h2> Presentations </h2> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Date</th> <th>Title</th> <th>Event</th> </tr> </thead> <tbody> <tr> <td>2016/11/29</td> <td>Automating Workflows for Analytics Pipelines(Sadayuki Furuhashi)(<a href="http://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-automating-workflows-for-analytics-pipelines-dev401">📊</a><a href="https://www.youtube.com/watch?v=odmTIZt4WFM">🎥</a>)</td> <td>AWS re:Invent 2016</td> </tr> <tr> <td>2017/06/01</td> <td>Automating Workflows for Analytics Pipelines(Sadayuki Furuhashi)<a href="https://www.slideshare.net/frsyuki/automating-workflows-for-analytics-pipelines">📊</a> </td> <td><a href="http://events.linuxfoundation.org/events/open-source-summit-japan">Open Source Summit</a></td> </tr> </tbody> </table></div> <h2> Blogs </h2> <ul> <li> <p>Scott Arbeitman @ Medium</p> <ul> <li><a href="https://medium.com/@scott_64558/in-praise-of-digag-an-alternative-to-airflow-258a0eef83bc">In Praise of Digdag, an Alternative to Airflow</a></li> </ul> </li> <li> <p>Thejesh GN</p> <ul> <li>2019-11-15 <a href="https://thejeshgn.com/2019/11/15/managing-secrets-in-digdag/">Managing Secrets in Digdag</a> </li> <li>2019-10-30 <a href="https://thejeshgn.com/2019/10/30/seven-useful-digdag-tips-and-tricks/">Seven Useful DigDag Tips and Tricks</a> </li> <li>2019-10-15 <a href="https://thejeshgn.com/2019/10/15/how-to-use-digdag-restful-apis-to-trigger-a-job/">How to use DigDag Restful APIs to Trigger a Job</a> </li> <li>2019-12-03 <a href="https://thejeshgn.com/2019/12/03/how-to-user-docker-to-run-workflows-in-digdag/">How to Use Docker to run Workflows in Digdag</a> </li> </ul> </li> <li> <p>Democratizing Data</p> <ul> <li>2019-10-06 <a href="https://blog.chezo.uno/how-to-test-a-new-docker-image-for-digdag-workflow-on-circleci-c8bb92987877">How to test a new Docker image for digdag workflow on CircleCI?</a> </li> <li>2020-02-05 <a href="https://chezo.uno/blog/py-operator-development-guide-for-python-users/">py&gt; operator development guide for Python users</a> </li> </ul> </li> <li> <p>dev.to</p> <ul> <li>Shubham Kadam <ul> <li>2020-02-15 <a href="https://dev.to/shubham_kadam/digdag-schedule-a-job-workflow-multiple-times-on-same-day-11g7">DigDag - Schedule a Job(workflow) multiple times on same day</a> </li> </ul> </li> </ul> </li> </ul> digdag hiroyuki sato Fri, 21 Jun 2019 04:29:55 +0000 https://dev.to/hiroysato/personal-link-collection-about-the-openucx-project-2e97 https://dev.to/hiroysato/personal-link-collection-about-the-openucx-project-2e97 <p>Personal link collection about the OpenUCX project.</p> <h2> OpenUCX </h2> <p><a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/http%3A%2F%2Fwww.openucx.org%2Fwp-content%2Fuploads%2F2015%2F07%2Fucx-architecture-1024x505.jpg" class="article-body-image-wrapper"><img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/http%3A%2F%2Fwww.openucx.org%2Fwp-content%2Fuploads%2F2015%2F07%2Fucx-architecture-1024x505.jpg"></a></p> <ul> <li> <a href="http://www.openucx.org" rel="noopener noreferrer">UCX Unified Communication X</a> Project page.</li> </ul> <h2> Slides・Videos </h2> <ul> <li>2019-06-19 <a href="https://youtu.be/1xpXzrV31Jw" rel="noopener noreferrer">Unified Communication X (UCX) Community - Gilad Shainer</a> OpenUCX Presentation in ISC19 <ul> <li><a href="https://youtu.be/1xpXzrV31Jw?t=281" rel="noopener noreferrer">UCX Development status</a></li> <li><a href="https://youtu.be/1xpXzrV31Jw?t=576" rel="noopener noreferrer">UCX Machine Layer in Charm++</a></li> <li> <a href="https://youtu.be/1xpXzrV31Jw?t=1067" rel="noopener noreferrer">UCX Support in MPICH</a> </li> <li><a href="https://youtu.be/1xpXzrV31Jw?t=1414" rel="noopener noreferrer">Enhancing MPI Communication using Accelerated Verbs and Tag Matching: The MVAPICH Approach</a></li> <li><a href="https://youtu.be/1xpXzrV31Jw?t=1749" rel="noopener noreferrer">UCX CUDA ROADMAP UPDATE</a></li> </ul> </li> <li>2019-06-19 <a href="http://www.openucx.org/wp-content/uploads/2019/06/UCX_ISC_2019_BoF_Final.pdf" rel="noopener noreferrer">OpenUCX Meeting at ISC#19</a> </li> <li>2018-03-21 <a href="https://on-demand.gputechconf.com/gtc/2019/video/_/S9679/" rel="noopener noreferrer">UCX-PYTHON: A FLEXIBLE COMMUNICATION LIBRARY FOR PYTHON APPLICATION</a> NVIDIA GTC</li> <li>2018-12-20 <a href="https://youtu.be/LLAV93cTyUI" rel="noopener noreferrer">UCX Presentation ISC’18</a> </li> <li>2017-12-06 <a href="https://youtu.be/hL542rj8jrI" rel="noopener noreferrer">UCX and UCF Projects for Exascale Move Forward at SC17</a> RichReport</li> <li>2016-02-26 <a href="https://youtu.be/HqgddTip86Y" rel="noopener noreferrer">UCX: An Open Source Framework for HPC Network APIs and Beyond ARM</a> Pavel Shamis from ARM</li> <li>2015-12-07 <a href="https://youtu.be/Cq0w7Kem9p4" rel="noopener noreferrer">UCX - Unified Communication X Framework - SC'15</a> Pavel Shamis, Oak Ridge National Lab</li> <li>2015-05-29 <a href="https://youtu.be/lD5CYMY840A" rel="noopener noreferrer">UCX - Unified Communication X Framework</a> RichReport</li> </ul> <h2> Related Projects. </h2> <ul> <li>2019-03-21 <a href="https://www.slideshare.net/MatthewRocklin/ucxpython-a-flexible-communication-library-for-python-applications" rel="noopener noreferrer">UCX-Python - A Flexible Communication Library for Python Applications</a> Matthew Rocklin, System Software Manager at NVIDIA</li> <li> <a href="https://docs.google.com/presentation/d/e/2PACX-1vSajAH6FzgQH4OwOJD5y-t9mjF9tTKEeljguEsfcjavp18pL4LkpABy4lW2uMykIUvP2dC-1AmhCq6l/pub?start=false&amp;loop=false&amp;delayms=60000&amp;slide=id.p1" rel="noopener noreferrer">High Performance Python Components</a>, Matthew Rocklin, System Software Manager at NVIDIA PASC 2019</li> <li>May 2019 <a href="https://charm.cs.illinois.edu/charmWorkshop/slides/26-Qin-UCX.pdf" rel="noopener noreferrer">The Effect of UCX Machine Layer on Charm++ Simulations</a> Yong Qin, Ph.D. Mellanox Technologies.</li> </ul> <h2> Papers. </h2> <ul> <li>Shamis, Pavel, Lopez, Matthew Graham, and Shainer, Gilad. Mon . <a href="https://www.osti.gov/biblio/1376625" rel="noopener noreferrer">"Enabling One-Sided Communication Semantics on ARM"</a>. United States. doi:10.1109/IPDPSW.2017.62. <a href="https://www.osti.gov/servlets/purl/1376625" rel="noopener noreferrer">https://www.osti.gov/servlets/purl/1376625</a>.</li> <li>Rohit Zambre, Megan Grodowitz, Aparna Chandramowlishwaran, and Pavel Shamis. 2019. <a href="https://dl.acm.org/citation.cfm?doid=3337821.3337910" rel="noopener noreferrer">Breaking Band: A Breakdown of High-performance Communication</a>. In Proceedings of the 48th International Conference on Parallel Processing (ICPP 2019). ACM, New York, NY, USA, Article 47, 10 pages. DOI: <a href="https://doi.org/10.1145/3337821.3337910" rel="noopener noreferrer">https://doi.org/10.1145/3337821.3337910</a> </li> </ul> <h2> Blogs </h2> <ul> <li>Towards Data Science <ul> <li>2019-06-29 <a href="https://towardsdatascience.com/python-performance-and-gpus-1be860ffd58d" rel="noopener noreferrer">Python, Performance, and GPUs A status update for using GPU accelerators from Python</a> </li> </ul> </li> </ul> <h2> Dev memo </h2> <h4> NBX, NB, and NBI </h4> <p><a href="https://twitter.com/pasha_sh/status/1298038366253047810?s=20" rel="noopener noreferrer">Twitter</a></p> <blockquote> <p>Implicit non blocking is “fire and forget” communication routine that does NOT return any request for tracking to the user. We will be deprecating NB and NBI APIs in the favor of NBX API that combines NB/NBI + extendable ABI.</p> </blockquote> <h2> Japanese </h2> <ul> <li>本当は怖い情報科学 <ul> <li>2018/02/25 <a href="http://freak-da.hatenablog.com/entry/2018/02/25/121759" rel="noopener noreferrer">UCXを試すメモ(1)</a> </li> </ul> </li> </ul> openucx hiroyuki sato Wed, 13 Dec 2017 10:16:13 +0000 https://dev.to/hiroysato/embulk-open-source-bulk-data-loader-documents-8j3 https://dev.to/hiroysato/embulk-open-source-bulk-data-loader-documents-8j3 <p><a href="http://www.embulk.org">Embulk</a> is an open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--mOW6-q3L--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://www.embulk.org/docs/_images/embulk-architecture.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--mOW6-q3L--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://www.embulk.org/docs/_images/embulk-architecture.png" alt=""></a></p> <p>I'm collecting <a href="http://embulk.org">Embulk</a> knowledge in the World. Please let me know if you find useful information.</p> <p>Hiroyuki Sato (twitter <a href="https://twitter.com/hiroysato">@hiroysato</a>)</p> <h2> Official page. </h2> <ul> <li> <a href="https://github.com/embulk/embulk">Official page</a>. </li> <li><a href="http://www.embulk.org/docs/">Document</a></li> </ul> <h2> Committer documents </h2> <ul> <li><a href="http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed">Embulk, an open-source plugin-based parallel bulk data loader</a></li> <li><a href="http://www.slideshare.net/frsyuki/embuk-internals">Embuk internals</a></li> <li> <a href="http://www.slideshare.net/frsyuki/embulk-56197273">Embulk Meetup keynote #2</a> Treasure Data, Inc. Sadayuki Furuhashi(@frsyuki)</li> </ul> <h2> Past events </h2> <ul> <li>2015-04-18 <a href="http://www.slideshare.net/repeatedly/fluentd-and-embulk-game-server-4">Fluentd and Embulk</a> Study for gaming server. Masahiro Nakagawa</li> <li>2015-07-31 <a href="http://www.slideshare.net/tagomoris/jruby-with-java-code-in-data-processing-world">JRuby with Java Code in Data Processing World</a> tagomoris <a href="http://2015.jrubyconf.eu">JRUBYCONF.eu</a> presentation <a href="https://www.youtube.com/watch?v=tPtELQyax0U">Video</a> </li> <li>2015-12-03 <a href="http://www.meetup.com/ja/awsgurus/events/226705053/?eventId=226705053">Loading data into Redshift - simplified with Schema-on-Read ELT</a> (<a href="http://www.slideshare.net/treasure-data/treasure-data-from-mysql-to-redshift">Slide</a>)</li> <li>2015-12-15 Embulk Meetup #2 <ul> <li> <a href="http://www.slideshare.net/frsyuki/embulk-56197273">Embulk Meetup #2 keynote</a> Treasure Data, Inc. Sadayuki Furuhashi (@frsyuki)</li> <li><a href="http://www.slideshare.net/oreradio/embulk-at-treasure-data">Embulk at Treasure Data, Inc.</a></li> </ul> </li> <li>2016-03-30 <a href="http://www.slideshare.net/muga.nishizawa/using-embulk-at-treasure-data">Using Embulk at Treasure Data</a> Treasure Data Inc. Muga Nishizawa Treasure Data Tech Talk in Japan.</li> <li>2016-05-03 <a href="http://www.slideshare.net/frsyuki/fighting-against-chaotically-separated-values-with-embulk">Fighting Against Chaotically Separated Values with Embulk</a> <a href="http://csvconf.com">csv,conf,v2</a> Treasure Data, Inc. Sadayuki Furuhashi (@frsyuki) (<a href="https://www.youtube.com/watch?v=RuA_SL5-sXY">Video</a>)</li> <li>2018-10-12 <a href="https://docs.google.com/presentation/d/1LxwJIA2BFbGtPtdaUaGp1mngYWUQDMrM1V3DZ8AneR0/edit#slide=id.p">Apache Airflow &amp; Embulk</a> (Antoine Augusti)</li> <li>2019-05-02 <a href="https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/74094">Learning how to perform ETL data migrations with open source tool Embulk</a> (Jason Bell) </li> </ul> <h2> Japanese documents. </h2> <p><a href="http://qiita.com/hiroysato/items/397f36c4838a0a93e352">Embulk documents in Japanese.</a></p> <h2> Others </h2> <ul> <li><a href="https://github.com/ccsd/canvas-data-embulk-configs">YAML configs for importing Canvas Data with Embulk</a></li> </ul> <h2> blog posts </h2> <ul> <li>Treasure Data <ul> <li><a href="https://docs.treasuredata.com/articles/embulk-overview">Embulk Overview</a></li> <li><a href="https://docs.treasuredata.com/articles/embulk-import-s3">Embulk for S3</a></li> <li><a href="https://docs.treasuredata.com/articles/embulk-import-mysql">Embulk for MySQL</a></li> <li><a href="https://docs.treasuredata.com/articles/embulk-import-googlesheet">Embulk for Google Sheets</a></li> </ul> </li> <li>DZone <ul> <li><a href="https://dzone.com/articles/how-to-succeed-at-importing-xml">How to Succeed at Importing XML</a></li> <li><a href="https://dzone.com/articles/weighing-bigquery-vs-redshift">Weighing BigQuery vs. Redshift</a></li> </ul> </li> <li>AWS Big Data Blog <ul> <li>July 14, 2016 <a href="https://blogs.aws.amazon.com/bigdata/post/Tx2V1BSKGITCMTU/How-SmartNews-Built-a-Lambda-Architecture-on-AWS-to-Analyze-Customer-Behavior-an">How SmartNews Built a Lambda Architecture on AWS to Analyze Customer Behavior and Recommend Content</a> </li> </ul> </li> <li>drivy.engineering <ul> <li>2017-12-11 <a href="https://drivy.engineering/embulk-data-transfer/">Embulk: move easily data across datasources</a> </li> </ul> </li> </ul> embulk java ruby etl