<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: TomXiaoYZ</title>
    <description>The latest articles on DEV Community by TomXiaoYZ (@tomxiaoyz).</description>
    <link>https://dev.to/tomxiaoyz</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1178219%2Fe44d943b-b568-488d-974a-cfe8df6dbefc.png</url>
      <title>DEV Community: TomXiaoYZ</title>
      <link>https://dev.to/tomxiaoyz</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/tomxiaoyz"/>
    <language>en</language>
    <item>
      <title>Claude Code plans, Codex builds: what 152 task cards taught me about governing two coding agents</title>
      <dc:creator>TomXiaoYZ</dc:creator>
      <pubDate>Tue, 17 Nov 2026 10:04:08 +0000</pubDate>
      <link>https://dev.to/tomxiaoyz/claude-code-plans-codex-builds-what-152-task-cards-taught-me-about-governing-two-coding-agents-2c2k</link>
      <guid>https://dev.to/tomxiaoyz/claude-code-plans-codex-builds-what-152-task-cards-taught-me-about-governing-two-coding-agents-2c2k</guid>
      <description>&lt;p&gt;I maintain several personal repositories: a data platform, a quant research repo with hard "never touch live trading" lines, a few e-commerce tools, and a content pipeline. Since late October I have split the work between two agents:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Claude Code defines and accepts.&lt;/strong&gt; It decides what to build, writes the task, and checks the result.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Codex implements.&lt;/strong&gt; It writes code and runs tests inside the spec it is given.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Between 2026-10-28 and 2026-11-14 this setup dispatched 152 task cards across 7 repositories. Four things moved it from "sometimes works" to "runs every day". Each one came out of a specific failure.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The handoff is a file in the repo, not a chat
&lt;/h2&gt;

&lt;p&gt;Early on I told Codex what to do in conversation. Context scattered, and when it finished I could not say whether it had drifted.&lt;/p&gt;

&lt;p&gt;Now the only handoff is a Markdown card at &lt;code&gt;docs/tasks/&amp;lt;date&amp;gt;-&amp;lt;topic&amp;gt;/README.md&lt;/code&gt;, committed to the task branch. Codex reads that card, and I review against the same commit. Every card has these sections:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Goal and user path&lt;/li&gt;
&lt;li&gt;Acceptance criteria (runnable commands or clickable pages)&lt;/li&gt;
&lt;li&gt;Boundaries&lt;/li&gt;
&lt;li&gt;An &lt;strong&gt;authorization&lt;/strong&gt; section (network, writable dirs, model, effort)&lt;/li&gt;
&lt;li&gt;Tests that must fail first&lt;/li&gt;
&lt;li&gt;A rounds log&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two of these sections earned their place.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Authorization is the only source of dispatch parameters.&lt;/strong&gt; The dispatcher reads permissions from the card and rejects command-line flags that would widen them. To widen scope, you write a new card. That way every escalation leaves a trace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I write the required tests first, and Codex must see them fail before implementing.&lt;/strong&gt; Behavior that no test covers is behavior Codex should not build.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Cards have a size cap and at most two rounds
&lt;/h2&gt;

&lt;p&gt;One early card grew to 700 lines across 4 files. Codex followed it, but I could not review it properly. A handoff card failed twice for environmental reasons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Round 1:&lt;/strong&gt; the worktree sat under the repo's &lt;code&gt;.codex/&lt;/code&gt; directory, which the sandbox treats as read-only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Round 2:&lt;/strong&gt; &lt;code&gt;git add&lt;/code&gt; was refused, because a linked worktree's &lt;code&gt;.git/&lt;/code&gt; metadata is also read-only to the sandbox.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both times Codex stopped, reported &lt;code&gt;blocked&lt;/code&gt; honestly, and touched nothing in the main checkout. The problem was the card and the environment, not the executor.&lt;/p&gt;

&lt;p&gt;So I added two hard limits:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Cards stay under 12 KB and contain no pasted source.&lt;/strong&gt; A card references paths and symbols, and Codex reads the worktree itself. A preflight script checks size, pasted code, missing sections, leftover placeholders, and line budgets over 300 without a stated reason.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two rounds maximum.&lt;/strong&gt; If round 2 is still returned, the card gets split or escalated to a human with evidence. No round 3.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Across the 152 cards:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;116 finished in one round (76%).&lt;/li&gt;
&lt;li&gt;29 took two rounds.&lt;/li&gt;
&lt;li&gt;7 took three or four:

&lt;ul&gt;
&lt;li&gt;1 predates the two-round rule.&lt;/li&gt;
&lt;li&gt;2 needed a browser or audio and video.&lt;/li&gt;
&lt;li&gt;4 broke the rule after it existed.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Those 4 cards show that "max two rounds" is ultimately enforced by a person. The summary script flags them, but it cannot stop someone from dispatching a third round.&lt;/p&gt;

&lt;p&gt;The browser cases produced another rule: &lt;strong&gt;threads started by my dispatcher cannot use the browser plugin's credentials.&lt;/strong&gt; So opening pages, taking screenshots and logging in stay on the Claude side, as acceptance steps.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. The manager costs 41x the engineer, by token volume
&lt;/h2&gt;

&lt;p&gt;This was the number I least expected. On the 141 cards where both sides could be measured:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Claude Code's median token volume (input plus cache) was 25.5M.&lt;/li&gt;
&lt;li&gt;Codex's was 0.8M.&lt;/li&gt;
&lt;li&gt;The median ratio was 41x, and the worst single card was 363x.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most of the Claude Code side is cache reads, which are much cheaper per token than fresh input, so 41x is a volume ratio, not a bill ratio. The direction is still clear: the expensive side is the one that plans and reviews and keeps growing its context.&lt;/p&gt;

&lt;p&gt;The reason is that every request resends the whole conversation. I started with one long session per project, and one transcript reached 32 MB. Late in that session, each turn re-read the history of every card before it.&lt;/p&gt;

&lt;p&gt;The fix was &lt;strong&gt;one card, one task session&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The project session only plans and writes cards.&lt;/li&gt;
&lt;li&gt;Dispatch, monitoring and acceptance happen in a fresh session that returns a verdict of 30 lines or fewer.&lt;/li&gt;
&lt;li&gt;The verdict is written into the card's rounds log, so the project session can be cleared without losing state.&lt;/li&gt;
&lt;li&gt;Acceptance runs in two stages: first write a checklist from the card alone, then read the diff against it. That keeps the reviewer from being anchored by the executor's own summary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The ratio is still high, and I am still working on it. The summary script now flags any card where the Claude Code side exceeds Codex by more than 5x.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The gate has bugs too
&lt;/h2&gt;

&lt;p&gt;I added a turn-end hook on the Codex side that audits the diff for budget overruns, files outside the card's list, and defensive code the card does not justify. If the audit fails, it sends Codex back within the same turn, at most twice.&lt;/p&gt;

&lt;p&gt;The hook then became a source of blocks itself. It mistook test function names for file paths, and it enforced a "tests ≤ 1.5x source" budget that no card had written down. Codex reported each case accurately in its receipt instead of deleting reasonable code to get past the audit.&lt;/p&gt;

&lt;p&gt;The lesson: &lt;strong&gt;a gate is code and has bugs.&lt;/strong&gt; In every review I now pull out "blocked by audit misfire" separately and fix the audit before blaming the executor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Take-aways
&lt;/h2&gt;

&lt;p&gt;The templates, preflight script and two-stage verdict format are in a sanitized example repo: &lt;a href="https://github.com/TomXiaoYZ/agent-harness-examples" rel="noopener noreferrer"&gt;https://github.com/TomXiaoYZ/agent-harness-examples&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Put the handoff in the repo and commit it to the branch.&lt;/li&gt;
&lt;li&gt;Cap card size, allow at most two rounds, then split or escalate.&lt;/li&gt;
&lt;li&gt;Measure cost per accepted change, not total usage. The expensive side is often the manager.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;strong&gt;Methodology.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scope:&lt;/strong&gt; local delegation evidence and Codex state DB, covering cards changed 2026-10-28 to 2026-11-14 (152 cards, 7 repos).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rounds:&lt;/strong&gt; the number of executor receipts per card.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Claude Code tokens:&lt;/strong&gt; input plus cache, summed over all Claude Code transcripts in that repo during the task window. This may include unrelated work, so it skews high.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Codex tokens:&lt;/strong&gt; the thread's &lt;code&gt;tokens_used&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>devops</category>
      <category>claude</category>
    </item>
    <item>
      <title>Data on EKS Platform</title>
      <dc:creator>TomXiaoYZ</dc:creator>
      <pubDate>Sun, 08 Oct 2023 07:13:56 +0000</pubDate>
      <link>https://dev.to/tomxiaoyz/data-on-eks-platform-51g8</link>
      <guid>https://dev.to/tomxiaoyz/data-on-eks-platform-51g8</guid>
      <description>&lt;h2&gt;
  
  
  &lt;strong&gt;Overview&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Since its launch in 2006, the Hadoop ecosystem has gained widespread acclaim. However, as big data components proliferate and usage increases, traditional big data platforms face growing challenges, including inefficient deployment of new components, limited resource sharing between applications, and ineffective operations and maintenance. As architectures remain static, this not only brings more complexity to operations, but also predictably high costs.&lt;/p&gt;

&lt;p&gt;Let me share our company’s experience as an example. Previously, our big data platform relied heavily on AWS cloud native services like EMR and EC2. Due to high costs, complex maintenance, and other issues, over the past six months we have gradually migrated all platform applications to EKS. This has reduced costs by 90%. During this time, data volumes have dropped 50% due to operational changes. By moving to EKS, we can now focus solely on building data applications, while basic ops are handled by our ops team for integrated management, reducing our investment in infrastructure.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--XlW_cMzG--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mt033rxhwnreh2afazdo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--XlW_cMzG--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mt033rxhwnreh2afazdo.png" alt="Image description" width="800" height="465"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We believe our experience could benefit any organization undertaking a containerization transition. Therefore, we plan to productize our big data platform and sell it on AWS Marketplace, making adoption easier.&lt;/p&gt;

&lt;p&gt;Our platform leverages Kubernetes and other cloud native technologies, providing a simplified architecture compared to traditional Hadoop. By offering this platform on AWS Marketplace, we aim to help organizations realize the benefits we have seen — 90% cost reduction and simplified ops — when shifting big data to containers on Kubernetes.&lt;/p&gt;

&lt;p&gt;With AWS handling infrastructure provisioning, customers can focus on data applications while our platform provides cluster deployment, autoscaling, monitoring, and more. We believe this turnkey approach will accelerate container adoption for big data use cases.&lt;/p&gt;

&lt;p&gt;Please reach out if you would like to learn more about our journey or get early access when we launch on AWS Marketplace. We welcome any feedback as we work to bring a proven big data container platform to the broader cloud native community.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Architecture&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--Teo0NlBt--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qajqxaf700033ugu5dkm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--Teo0NlBt--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qajqxaf700033ugu5dkm.png" alt="Image description" width="800" height="469"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Data on EKS Platform (DEP) is a Kubernetes-based cloud-native big data platform developed by our company. Built using open source components, DEP integrates Hive, Spark, Flink, Trino, HDFS, Kafka and more. It provides data development and BI platforms including Superset, Metabase, Jupyter, Zeppelin and custom IDE platform. DEP’s metadata management and data quality monitoring system improve data quality, while data ops and user profiling enable data-driven capabilities. Ops management and permissions ensure security and stability.&lt;/p&gt;

&lt;p&gt;DEP is an all-in-one platform enabling admins, developers, analysts and business users to complete daily tasks. Leveraging Kubernetes auto-scaling, self-healing and simplified ops, DEP provides a managed big data environment. Companies can focus on data while DEP handles infrastructure, monitoring, access controls and more. Compared to legacy Hadoop, DEP reduces costs while empowering employees through faster, easier analytics and insights access.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Product Features&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;DEP’s main functions include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;DEP Core, which contains scheduling center, IDE, metadata management, and data quality monitoring capabilities.&lt;/li&gt;
&lt;li&gt;Nebula, the data requirement module, which helps users easily find, understand, request access to datasets and raise new data requests to data engineers or data analysts.&lt;/li&gt;
&lt;li&gt;Data-driven capabilities including data operations and user profiling to generate insights.&lt;/li&gt;
&lt;li&gt;New features currently in development:&lt;/li&gt;
&lt;li&gt;Model development and deployment to support machine learning workflows. Using models built by data scientists in production applications and at scale on DEP.&lt;/li&gt;
&lt;li&gt;Large language model applications to allow conversational interaction with data and insights using chatbots and virtual assistants.
DEP aims to provide an integrated environment covering the full data lifecycle. Users can discover datasets, develop models and applications, deploy them at scale, monitor data quality, and enable self-service analytics. The platform leverages Kubernetes to provide flexibility, scalability, and automated management. We welcome customer feedback to prioritize additional capabilities that would maximize the value of DEP.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Scheduling Center&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;On DEP, we developed a scheduling center to handle thousands of different types of scheduling tasks across the platform. The scheduling center currently supports the following job types:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SYNC: Extract data from business databases (MySQL, PostgreSQL, etc) into the data warehouse ODS layer. Supports full or incremental extracting, customizable data cleansing rules, and partitioning.&lt;/li&gt;
&lt;li&gt;HIVE,SPARK,TRINO: Supports SQL jobs using different compute engines.&lt;/li&gt;
&lt;li&gt;JAVA: Supports execution of custom Jar packages.&lt;/li&gt;
&lt;li&gt;SHELL: Supports execution of other custom jobs.&lt;/li&gt;
&lt;li&gt;EXPORT: Exports ADS layer data back to business databases for online business queries.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--_1CsfJK3--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhdt737p6zg772te55u3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--_1CsfJK3--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhdt737p6zg772te55u3.png" alt="Image description" width="800" height="478"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--JAePiZqc--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dm3yzi1la5ikmdlb537e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--JAePiZqc--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dm3yzi1la5ikmdlb537e.png" alt="Image description" width="800" height="121"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--3cm8JeJz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/09ku3pp33hpyvsfwnwx5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--3cm8JeJz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/09ku3pp33hpyvsfwnwx5.png" alt="Image description" width="800" height="346"&gt;&lt;/a&gt;&lt;br&gt;
Additionally, the scheduling center supports automatic lineage determination, DAG diagrams, job logging, and job alerts.&lt;/p&gt;

&lt;p&gt;The scheduling center provides a unified interface to develop, monitor, and manage ETL pipelines, data integration, and processing workloads. By leveraging Kubernetes concepts like controllers, the system is scalable, resilient, and automates repetitive tasks. We plan to continue enhancing the scheduling center with machine learning capabilities for intelligent optimization and automation.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;IDE&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;On DEP, we developed a custom IDE to provide ad-hoc querying, data development, and job approval workflows.&lt;/p&gt;

&lt;p&gt;In our IDE, data developers can use Hive or Trino SQL to develop, test, and request deployment of jobs. Business analysts can query data themselves and download data once approved. Every query goes through Ranger and Solr for data authorization and auditing.&lt;/p&gt;

&lt;p&gt;The IDE provides a collaborative environment for different user personas with built-in governance. Data engineers have the tools to develop pipelines and ETL jobs. Analysts have self-service access to data for exploratory analysis. Queries are logged and access controlled to meet security policies.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--nNOWBtL8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kwdxerg4sbg8lvbaphj6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--nNOWBtL8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kwdxerg4sbg8lvbaphj6.png" alt="Image description" width="800" height="455"&gt;&lt;/a&gt;&lt;br&gt;
Key benefits include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Unified interface for SQL, development, testing before deploying to production&lt;/li&gt;
&lt;li&gt;Collaboration between engineers and analysts&lt;/li&gt;
&lt;li&gt;Data discovery for business users&lt;/li&gt;
&lt;li&gt;Approval workflows for access requests&lt;/li&gt;
&lt;li&gt;Centralized data authorization, auditing, and lineage tracking
By leveraging the IDE, DEP customers can accelerate analytics development, empower users with self-service, and maintain oversight — all through a single pane of glass. We welcome additional feedback on IDE capabilities that would further enhance the user experience.
&lt;strong&gt;Metadata Management &amp;amp; Data Quality&lt;/strong&gt;
Metadata management is a crucial process in building a data platform, enabling enterprises to maximize the value of their data assets. Metadata management helps business analysts, architects, data engineers, developers and other stakeholders clearly understand what data an organization has, where it is stored, how to extract, cleanse, maintain it, and guide users.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s---izivecz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dd88xyx7pli7fkw0cpw8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s---izivecz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dd88xyx7pli7fkw0cpw8.png" alt="Image description" width="800" height="237"&gt;&lt;/a&gt;&lt;br&gt;
A key benefit of metadata management is ensuring data quality. In DEP, we provide additional data quality monitoring capabilities, calculating metrics and alerts at the table, column and custom rule levels. This allows data teams to promptly discover and address anomalies.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--69j2-rEb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/15srlupfphfks8y6goxc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--69j2-rEb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/15srlupfphfks8y6goxc.png" alt="Image description" width="800" height="287"&gt;&lt;/a&gt;&lt;br&gt;
With DEP’s metadata management, companies can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Document data sources, schemas, ETL pipelines, dependencies&lt;/li&gt;
&lt;li&gt;Discover data to understand availability across silos&lt;/li&gt;
&lt;li&gt;Track lineage to trust data provenance&lt;/li&gt;
&lt;li&gt;Manage governance policies like security, access&lt;/li&gt;
&lt;li&gt;Monitor data quality with automated alerts&lt;/li&gt;
&lt;li&gt;Guide and train end users on datasets
Strong metadata management and data quality monitoring provides the foundation for reliable analytics, trusted insights, and data democratization. We welcome additional perspectives on how DEP can further enhance metadata and governance capabilities to maximize value.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Request Center&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;We developed Request Center, a data requirements platform, to manage business demands. Business users can submit requests for data applications, dashboards, and data warehouse development. Data developers can promptly follow up and process these requests on the platform.&lt;/p&gt;

&lt;p&gt;The data requirements platform strengthens the connection between business and data teams, reducing communication barriers.&lt;/p&gt;

&lt;p&gt;Key benefits include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Central platform for data requests across the organization&lt;/li&gt;
&lt;li&gt;Standardized intake forms and workflows&lt;/li&gt;
&lt;li&gt;Transparency into request status and handling&lt;/li&gt;
&lt;li&gt;Prioritization based on business needs&lt;/li&gt;
&lt;li&gt;Collaboration between business and technical teams&lt;/li&gt;
&lt;li&gt;Audit trail of requirements gathering and delivery&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--XKK9Nfrf--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/srdhjukssgogaequi7dr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--XKK9Nfrf--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/srdhjukssgogaequi7dr.png" alt="Image description" width="643" height="614"&gt;&lt;/a&gt;&lt;br&gt;
Request Center allows business users to easily log and track requests, ensuring their needs are captured and worked on. For data teams, it provides a single workflow to manage incoming tasks rather than ad-hoc emails and messages.&lt;/p&gt;

&lt;p&gt;By streamlining data requirements gathering through the Request Center, organizations can align business stakeholders and data teams, accelerate analytics delivery, and use data more strategically. We welcome additional input on how to further enhance and leverage the platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Other Features&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Data Operations &amp;amp; User Profiling&lt;/strong&gt;&lt;br&gt;
Empowering business users to apply data in reasonable and rapid ways to feed online business operations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Permission Center&lt;/strong&gt;&lt;br&gt;
On DEP, we have defined hundreds of granular permissions that can be combined to assign appropriate roles to members.&lt;/p&gt;

&lt;p&gt;For example, data engineers may have access to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Create/edit tables, views in Hive&lt;/li&gt;
&lt;li&gt;Develop ETL jobs with Sqoop, Spark, etc&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Schedule and run jobs in production&lt;br&gt;
Analysts may have permissions like:&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Read access to certain tables and views&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Ability to run ad-hoc queries in the IDE&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Export limited results sets&lt;br&gt;
Fine-grained authorization allows implementing least privilege access tailored to each user’s responsibilities. Complex permissions can be abstracted into reusable roles mapped to job functions.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Key benefits include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Align access with organizational data security policies&lt;/li&gt;
&lt;li&gt;Prevent unauthorized or unintentional data access&lt;/li&gt;
&lt;li&gt;Simplify user management by bundling permissions into roles&lt;/li&gt;
&lt;li&gt;Gain visibility into who can access what data&lt;/li&gt;
&lt;li&gt;Streamline auditing for compliance&lt;/li&gt;
&lt;li&gt;DEP’s flexible and granular permission model balances security with enabling self-service analytics.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Configuration Center&lt;/strong&gt;&lt;br&gt;
On DEP, we provide platform configuration capabilities through a Parameter Center where data administrators can manage table naming conventions, job labels, table tags, and other custom rules. Configurations take effect in the system in real-time.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Product Timeline&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--CQrYrzNn--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/se3306kuknm6m6j9u1m3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--CQrYrzNn--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/se3306kuknm6m6j9u1m3.png" alt="Image description" width="800" height="720"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Pricing&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Pricing Models&lt;/strong&gt;&lt;br&gt;
We offer annual and monthly subscription plans with one-click deployment through AWS CloudFormation for a ready-to-use solution.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Subscription Plans&lt;/strong&gt;&lt;br&gt;
We provide a unified edition for all customers. Monthly fee is $800 and annual fee is $8,000.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Value-Added Services&lt;/strong&gt;&lt;br&gt;
We provide additional services like training, technical support, and can customize exclusive solutions based on user needs.&lt;/p&gt;

&lt;p&gt;Key pricing details:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Predictable subscription model avoids variable cloud costs&lt;/li&gt;
&lt;li&gt;Annual contract offers discounted rate with one yearly payment&lt;/li&gt;
&lt;li&gt;Monthly subscriptions allow flexibility to start small and expand&lt;/li&gt;
&lt;li&gt;Simple per-node pricing, no hidden fees or extra charges&lt;/li&gt;
&lt;li&gt;Add-on services available for custom integrations, training, support&lt;/li&gt;
&lt;li&gt;By leveraging our AWS Marketplace product, customers can deploy DEP quickly with minimal upfront costs. We aim to provide a high-value and scalable platform enabling organizations to realize fast time-to-value. Please reach out for customized pilots, proofs of concept, or pricing discussions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For more information about DEP, please feel free to contact &lt;a href="mailto:tom.xiao@hotmail.com"&gt;tom.xiao@hotmail.com&lt;/a&gt; if you are interested.&lt;/p&gt;

&lt;p&gt;We will provide updates on the product development progress and usage instructions through additional blogs posts and announcements.&lt;/p&gt;

&lt;p&gt;We welcome you to try out a demo, sign up for early access once launched on AWS Marketplace, or engage in a pilot project to see how DEP can empower your data teams and organization.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Recovering Accidentally Transitioned S3 Files from the Glacier Tier</title>
      <dc:creator>TomXiaoYZ</dc:creator>
      <pubDate>Sun, 08 Oct 2023 07:03:34 +0000</pubDate>
      <link>https://dev.to/tomxiaoyz/recovering-accidentally-transitioned-s3-files-from-the-glacier-tier-1nc1</link>
      <guid>https://dev.to/tomxiaoyz/recovering-accidentally-transitioned-s3-files-from-the-glacier-tier-1nc1</guid>
      <description>&lt;p&gt;Are you stuck with S3 files that were accidentally moved to the Glacier storage tier? Do you need an efficient yet simple way to restore them? If yes, then this guide is for you.&lt;/p&gt;

&lt;p&gt;Amazon S3 is a scalable object storage service while Amazon Glacier is a secure, durable, and low-cost storage service for data archiving and long-term backup. Sometimes, files may end up in Glacier storage either intentionally, for cost-saving purposes, or unintentionally due to errors.&lt;/p&gt;

&lt;p&gt;Regardless, the restoration process can be daunting. This guide will help you simplify this process with Python and AWS’s boto3 library.&lt;/p&gt;

&lt;p&gt;Below is the Python script that will get the job done:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;import multiprocessing as mp
import os
import sys
import shutil
import traceback
import zipfile
import requests
import json
import socket
import boto3
import threadpool
import time
from datetime import timedelta, datetime
from pymongo import MongoClient
from loguru import logger
from botocore.exceptions import ClientError
import re

ACK = "Your Access Key"
ACS = "Your Secret Key"

bucket_name = "Your Bucket Name"
s3_remote_dir = "Path to Your S3 Directory"

s3 = boto3.client('s3',  aws_access_key_id=ACK, aws_secret_access_key=ACS)

# local path
local_save_path = './temp/'  # temporarily download files to local path


def _get_all_s3_objects(**base_kwargs):
    """
    Get all objects under s3_remote_dir
    """
    try:
        continuation_token = None
        while True:
            list_kwargs = dict(MaxKeys=1000, **base_kwargs)
            if continuation_token:
                list_kwargs['ContinuationToken'] = continuation_token
            response = s3.list_objects_v2(**list_kwargs)
            yield from response.get('Contents', [])
            if not response.get('IsTruncated'):  # At the end of the list?
                break
            continuation_token = response.get('NextContinuationToken')
    except:
        # send_dingtalk_message(traceback.format_exc())
        logger.error(traceback.format_exc())


def head_object(bucket_name, object_name):
    s3 = boto3.client('s3')
    response = None
    try:
        response = s3.head_object(Bucket=bucket_name, Key=object_name)
    except ClientError as e:
        logger.error(e)
        logger.error(
            f"NoSuchBucket, NoSuchKey, or InvalidObjectState error == the object's, storage class was not GLACIER. {bucket_name} {object_name} ")
        return None
    return response


# resore objects from glacier tier
def restore_object(bucket_name, object_name, days, retrieval_type='Expedited'):
    request = {'Days': days,
                'GlacierJobParameters': {'Tier': retrieval_type}}
    # s3 = boto3.client('s3')
    try:
        s3.restore_object(Bucket=bucket_name, Key=object_name, RestoreRequest=request)
    except ClientError as e:
        logger.error(e)
        logger.error(
            f"NoSuchBucket, NoSuchKey, or InvalidObjectState error == the object's, storage class was not GLACIER. {bucket_name} {object_name} ")

        return False
    return True


key_content_list = []
total_content_size = 0


while True:
    doing_count, done_count, need_count = 0, 0, 0

    s3_objects = _get_all_s3_objects(Bucket=bucket_name, Prefix=s3_remote_dir)
    for obj in s3_objects:
        key = obj.get('Key', None)
        file_name = key.split("/")[-1]
        # In hive, files under glacier tier are then moved to sub-directories called 'HIVE_UNION_SUBDIR_*'
        # We need to move it to the right place
        to_key = re.sub("\/HIVE_UNION_SUBDIR_[\d]+\/", "/", key)
        print(key, to_key, file_name)
        success = head_object(bucket_name, key)
        need_count += 1

        if success:
            if success.get('Restore'):
                print('Restore {}'.format(success['Restore']))
                index = success['Restore'].find('ongoing-request=\"false\"')
                if -1 == index:
                    print(f"{need_count} Under recovering...{key}")
                    doing_count += 1
                else:
                    print(success['Restore'][success['Restore'].find('expiry-date='):])
                    print(f"{need_count} Recover succeeded...{key}")
                    find_spec_path = file_name.find('HIVE_UNION_SUBDIR')
                    if -1 != find_spec_path:
                        print("no need to download...{key}")
                    else:
                        s3.download_file(bucket_name, key, local_save_path + file_name)
                        s3.upload_file(local_save_path + file_name, bucket_name, to_key)
                    done_count += 1
            else:
                print(f'{need_count} neet to recovery... {key}')
                restore_object(bucket_name, key, 10)
        print(doing_count, done_count, need_count)

    if done_count == need_count:
        break
    time.sleep(15)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Understanding the Code&lt;/strong&gt;&lt;br&gt;
The solution uses the boto3 library to interact with the AWS S3 and Glacier services.&lt;/p&gt;

&lt;p&gt;The _get_all_s3_objects function is created to list all the objects in the specified S3 bucket and directory.&lt;/p&gt;

&lt;p&gt;The head_object function retrieves metadata from an object without returning the object itself. This is useful for checking if a file exists and its status.&lt;/p&gt;

&lt;p&gt;The restore_object function is the heart of the script. It initiates a job to restore a file from Glacier to S3. The Days parameter specifies the lifetime of the temporary copy of the object in the S3 bucket, while GlacierJobParameters sets the speed (tier) of the restoration process.&lt;/p&gt;

&lt;p&gt;The script then loops through the objects in the specified S3 location. For each object, it checks if a restore operation is already in progress or completed using the head_object function. If a restore operation is in progress, it leaves it alone. If the restore operation is complete, it re-uploads the file to the desired location. If the file hasn't been restored at all, it starts a restore operation.&lt;/p&gt;

&lt;p&gt;The process continues until all files have been restored.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In Conclusion&lt;/strong&gt;&lt;br&gt;
The script provides a way to automate the process of restoring files from the Glacier storage tier back to S3. It can be a lifesaver if you’ve got hundreds or even thousands of files to restore.&lt;/p&gt;

&lt;p&gt;Remember to replace the placeholders in the script with your actual AWS credentials, bucket names, and file paths. Happy restoring!&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Setup HTTPs Authentication for Presto / Trino</title>
      <dc:creator>TomXiaoYZ</dc:creator>
      <pubDate>Sun, 08 Oct 2023 06:55:04 +0000</pubDate>
      <link>https://dev.to/tomxiaoyz/setup-https-authentication-for-presto-trino-30ag</link>
      <guid>https://dev.to/tomxiaoyz/setup-https-authentication-for-presto-trino-30ag</guid>
      <description>&lt;p&gt;Version: Presto 0.261 / Trino 360&lt;/p&gt;

&lt;p&gt;In this article, we are going to enable HTTPS-based front-end password authentication for the Presto CLI or JDBC and ODBC drivers. The Presto CLI sends the username and password to the Presto coordinator, who then validates those credentials using an external LDAP service or user password store file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: Generate TLS Java Keystore File
&lt;/h2&gt;

&lt;p&gt;When using Kerberos and LDAP authentication, the Presto coordinator must be accessed via HTTPS. The Presto coordinator uses Java Keystore files for TLS configuration. These keys are generated using keytool and stored in the Java Keystore file of the Presto coordinator.&lt;br&gt;
&lt;code&gt;keytool -genkeypair -alias presto_keystore -keyalg RSA -keystore /etc/presto_keystore.jks&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--7HEqVI30--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l7fdl1qu512yw7asfhgn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--7HEqVI30--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l7fdl1qu512yw7asfhgn.png" alt="Image description" width="800" height="243"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Note: Please Make Sure CN is your IP address!&lt;/strong&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Step 2: Generate TLS Java Truststore File
&lt;/h2&gt;

&lt;p&gt;The truststore file contains the certificate of a trusted TLS/SSL server or a certificate of a trusted certification authority that identifies the server.&lt;/p&gt;

&lt;p&gt;To ensure secure access to the Presto coordinator over HTTPS, clients can configure a truststore.&lt;/p&gt;

&lt;p&gt;In order for the Presto CLI to trust the Presto coordinator, the coordinator’s certificate must be imported into the CLI’s truststore.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;keytool -exportcert -keystore /etc/presto_keystore.jks -alias presto_keystore -file /etc/alternatives/jre/lib/security/presto_certificate.cer -storepass {your password}
keytool -import -alias presto_trust -keystore /etc/alternatives/jre/lib/security/cacerts -file /etc/alternatives/jre/lib/security/presto_certificate.cer --storepass changeit -trustcacerts
# changeit is the default password for Java truststore
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Step 3: Install a trusted certificate to the client in code
&lt;/h2&gt;

&lt;p&gt;vim ~/InstallCert.java&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/*
 * Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Sun Microsystems nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.security.KeyStore;
import java.security.MessageDigest;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;

public class InstallCert {


 public static void main(String[] args) throws Exception {

  String host;
  int port;
  char[] passphrase;
  if ((args.length == 1) || (args.length == 2)) {

   String[] c = args[0].split(":");
   host = c[0];
   port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);
   String p = (args.length == 1) ? "changeit" : args[1];
   passphrase = p.toCharArray();
  } else {

   System.out
     .println("Usage: java InstallCert &amp;lt;host&amp;gt;[:port] [passphrase]");
   return;
  }

  File file = new File("jssecacerts");
  if (file.isFile() == false) {

   char SEP = File.separatorChar;
   File dir = new File(System.getProperty("java.home") + SEP + "lib"
     + SEP + "security");
   file = new File(dir, "jssecacerts");
   if (file.isFile() == false) {

    file = new File(dir, "cacerts");
   }
  }
  System.out.println("Loading KeyStore " + file + "...");
  InputStream in = new FileInputStream(file);
  KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
  ks.load(in, passphrase);
  in.close();

  SSLContext context = SSLContext.getInstance("TLS");
  TrustManagerFactory tmf = TrustManagerFactory
    .getInstance(TrustManagerFactory.getDefaultAlgorithm());
  tmf.init(ks);
  X509TrustManager defaultTrustManager = (X509TrustManager) tmf
    .getTrustManagers()[0];
  SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);
  context.init(null, new TrustManager[] {
             tm }, null);
  SSLSocketFactory factory = context.getSocketFactory();

  System.out
    .println("Opening connection to " + host + ":" + port + "...");
  SSLSocket socket = (SSLSocket) factory.createSocket(host, port);
  socket.setSoTimeout(10000);
  try {

   System.out.println("Starting SSL handshake...");
   socket.startHandshake();
   socket.close();
   System.out.println();
   System.out.println("No errors, certificate is already trusted");
  } catch (SSLException e) {

   System.out.println();
   e.printStackTrace(System.out);
  }

  X509Certificate[] chain = tm.chain;
  if (chain == null) {

   System.out.println("Could not obtain server certificate chain");
   return;
  }

  BufferedReader reader = new BufferedReader(new InputStreamReader(
    System.in));

  System.out.println();
  System.out.println("Server sent " + chain.length + " certificate(s):");
  System.out.println();
  MessageDigest sha1 = MessageDigest.getInstance("SHA1");
  MessageDigest md5 = MessageDigest.getInstance("MD5");
  for (int i = 0; i &amp;lt; chain.length; i++) {

   X509Certificate cert = chain[i];
   System.out.println(" " + (i + 1) + " Subject "
     + cert.getSubjectDN());
   System.out.println("   Issuer  " + cert.getIssuerDN());
   sha1.update(cert.getEncoded());
   System.out.println("   sha1    " + toHexString(sha1.digest()));
   md5.update(cert.getEncoded());
   System.out.println("   md5     " + toHexString(md5.digest()));
   System.out.println();
  }

  System.out
    .println("Enter certificate to add to trusted keystore or 'q' to quit: [1]");
  String line = reader.readLine().trim();
  int k;
  try {

   k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;
  } catch (NumberFormatException e) {

   System.out.println("KeyStore not changed");
   return;
  }

  X509Certificate cert = chain[k];
  String alias = host + "-" + (k + 1);
  ks.setCertificateEntry(alias, cert);

  OutputStream out = new FileOutputStream("jssecacerts");
  ks.store(out, passphrase);
  out.close();

  System.out.println();
  System.out.println(cert);
  System.out.println();
  System.out
    .println("Added certificate to keystore 'jssecacerts' using alias '"
      + alias + "'");
 }

 private static final char[] HEXDIGITS = "0123456789abcdef".toCharArray();

 private static String toHexString(byte[] bytes) {

  StringBuilder sb = new StringBuilder(bytes.length * 3);
  for (int b : bytes) {

   b &amp;amp;= 0xff;
   sb.append(HEXDIGITS[b &amp;gt;&amp;gt; 4]);
   sb.append(HEXDIGITS[b &amp;amp; 15]);
   sb.append(' ');
  }
  return sb.toString();
 }

 private static class SavingTrustManager implements X509TrustManager {


  private final X509TrustManager tm;
  private X509Certificate[] chain;

  SavingTrustManager(X509TrustManager tm) {

   this.tm = tm;
  }

  public X509Certificate[] getAcceptedIssuers() {

   throw new UnsupportedOperationException();
  }

  public void checkClientTrusted(X509Certificate[] chain, String authType)
    throws CertificateException {

   throw new UnsupportedOperationException();
  }

  public void checkServerTrusted(X509Certificate[] chain, String authType)
    throws CertificateException {

   this.chain = chain;
   tm.checkServerTrusted(chain, authType);
  }
 }

}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compile InstallCert.java and run&lt;/p&gt;

&lt;p&gt;&lt;code&gt;javac InstallCert.java&lt;br&gt;
java InstallCert {coordinator hostname} or&lt;br&gt;
java InstallCert {coordinator hostname}:port or&lt;br&gt;
java InstallCert {coordinator hostname}:port password&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Enter 1, enter, and generate a certificate named jssecacerts in the current directory. Copy the certificate to the $JAVA_HOME/jre/lib/security directory&lt;/p&gt;

&lt;p&gt;&lt;code&gt;mv jssecacerts /etc/alternatives/jre/lib/security/&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4. Repeat Step 1~3 on each of your worker node. Remember to replace IP in Step 1 with current node IP.
&lt;/h2&gt;

&lt;h2&gt;
  
  
  Step 5. Configure Presto/Trino config.properties.
&lt;/h2&gt;

&lt;p&gt;Remain the same on each of your coordinator and worker node.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;discovery.uri=http://ip-10-186-17-52.ap-northeast-1.compute.internal:9002&lt;br&gt;
http-server.authentication.type = PASSWORD,CERTIFICATE&lt;br&gt;
http-server.https.enabled = true&lt;br&gt;
node.internal-address-source = FQDN&lt;br&gt;
http-server.http.enabled = true&lt;br&gt;
http-server.https.port = 8889&lt;br&gt;
http-server.http.port = 9002&lt;br&gt;
http-server.https.keystore.path = /etc/presto_keystore.jks&lt;br&gt;
http-server.https.keystore.key = {replace with you keystore password}&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Create an password-authenticator.properties file on the coordinator&lt;/p&gt;

&lt;p&gt;Here you can configure the use of LDAP or password file authentication. We will use password file authentication.&lt;/p&gt;

&lt;p&gt;For more configuration details, please refer to the official website: &lt;a href="https://prestodb.io/docs/current/security/ldap.html"&gt;https://prestodb.io/docs/current/security/ldap.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;password-authenticator.name=file &lt;br&gt;
file.password-file=/etc/presto/conf/password.db&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 6. Create Password File
&lt;/h2&gt;

&lt;p&gt;Create an empty password file to get started:&lt;br&gt;
&lt;code&gt;touch password.db&lt;/code&gt;&lt;br&gt;
Add or update password for user ‘test’&lt;br&gt;
&lt;code&gt;htpasswd -B -C 10 password.db test&lt;/code&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 7. Restart Presto/Trino
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;systemctl restart presto-server&lt;br&gt;
systemctl restart trino-server&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Congratulation!&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
