DEV Community: Arun Kumar

Building Ethical AI: A Comprehensive Guide to Responsible Artificial Intelligence

Arun Kumar — Fri, 03 Oct 2025 13:55:21 +0000

Introduction

Ethical AI encompasses methodologies and frameworks designed to create AI systems that maintain transparency and reliability while reducing potential hazards and adverse impacts. These ethical standards must be integrated across the complete AI application journey, spanning initial conception, creation, implementation, oversight, and assessment stages.

Core Principles for Ethical AI Implementation

Organizations seeking to implement AI ethically should proactively establish systems that are:

• Completely transparent and answerable, incorporating supervision and governance frameworks
• Overseen by executive leadership responsible for ethical AI strategies
• Created by teams possessing deep knowledge in ethical AI methodologies and applications
• Constructed according to established ethical AI frameworks

Understanding Generative AI and Foundation Models

Generative artificial intelligence operates through foundation models (FMs) - sophisticated systems pre-trained on vast collections of general-purpose data extending far beyond proprietary datasets. These versatile models can execute diverse functions and, when provided with user instructions (typically text-based prompts), produce original content by leveraging learned patterns and correlations to anticipate optimal outputs.

Common applications of generative AI encompass conversational agents, automated code creation, and text-to-image synthesis.

Addressing Accuracy Challenges in AI Systems

The primary obstacle confronting AI developers is achieving reliable accuracy. Both conventional and generative AI solutions rely on models trained using specific datasets, limiting their predictive and generative capabilities to their training scope. Inadequate training protocols inevitably produce unreliable outcomes, making it crucial to tackle bias and variance within models.

Understanding Bias in AI Models

Bias represents a fundamental challenge in AI development, occurring when models fail to capture essential characteristics within datasets due to oversimplified data representation. Bias is quantified by measuring discrepancies between model predictions and actual target values.
Minimal differences indicate low bias, while substantial gaps suggest high bias conditions. High-bias models suffer from underfitting - failing to recognize sufficient data feature variations, resulting in poor training performance.

Managing Variance in Machine Learning

Variance presents distinct developmental challenges, describing a model's susceptibility to training data fluctuations and noise. Problematically, models may interpret data noise as significant output factors. Elevated variance causes models to become overly familiar with training datasets, achieving high training accuracy by capturing all data characteristics. However, when exposed to novel data with different features, accuracy deteriorates significantly. This creates overfitting scenarios where models excel on training data but fail on evaluation datasets due to memorization rather than generalization capabilities.

Unique Challenges in Generative AI

While generative AI offers distinctive advantages, it also presents specific challenges including content toxicity, hallucinations, intellectual property concerns, and academic integrity issues.

Content Toxicity

Toxicity involves generating inappropriate, offensive, or disturbing content across various media formats. This represents a primary generative AI concern, complicated by the difficulty of defining and scoping toxic content. Subjective interpretations of toxicity create additional challenges, with boundaries between content restriction and censorship remaining contextually and culturally dependent. Technical difficulties include identifying subtly offensive content that avoids obviously inflammatory language.

AI Hallucinations

Hallucinations manifest as plausible-sounding but factually incorrect assertions. Given the probabilistic word prediction methods employed by large language models, hallucinations are particularly problematic in factual applications. A common example involves LLMs generating fictitious academic citations when prompted about specific authors' publications, creating realistic-seeming but entirely fabricated references.

Intellectual Property Protection

Early LLMs occasionally reproduced verbatim training data passages, raising privacy and legal concerns. While improvements have addressed direct copying, more nuanced content reproduction remains problematic. For instance, requesting generative image models to create artwork "in the style of" famous artists raises questions about artistic mimicry and originality.

Academic Integrity Concerns

Generative AI's creative capabilities raise concerns about misuse in academic and professional contexts, including essay writing and job application materials. Educational institutions maintain varying perspectives, with some prohibiting generative AI use in evaluated content while others advocate adapting educational practices to embrace new technologies. The fundamental challenge of verifying human authorship will likely persist across multiple contexts.

Professional Impact and Transformation

Generative AI's proficiency in creating compelling content, performing well on standardized assessments, and producing comprehensive articles has generated concerns about professional displacement. While premature predictions should be avoided, generative AI will likely transform many work aspects, potentially automating previously human-exclusive tasks.

Essential Dimensions of Responsible AI

Responsible AI encompasses multiple interconnected dimensions: fairness, explainability, privacy protection, security, robustness, governance, transparency, safety, and controllability. These elements function as integrated components rather than standalone objectives, requiring comprehensive implementation for complete responsible AI achievement.

AWS Tools for Responsible AI Implementation

As a cloud technology leader, AWS provides services including Amazon SageMaker AI and Amazon Bedrock with integrated responsible AI tools. These platforms address foundation model evaluation, generative AI safeguards, bias detection, prediction explanations, monitoring capabilities, human review processes, and governance enhancement.

Foundation Model Evaluation

Organizations should thoroughly evaluate foundation models for specific use case suitability. Amazon offers evaluation capabilities through Amazon Bedrock and Amazon SageMaker AI Clarify.

Amazon Bedrock Model Evaluation enables foundation model evaluation, comparison, and selection through simple interfaces, offering both automatic and human evaluation options:

• Automatic evaluation: Provides predefined metrics including accuracy, robustness, and toxicity assessment
• Human evaluation: Addresses subjective metrics such as friendliness, style, and brand alignment using internal teams or AWS-managed reviewers

SageMaker AI Clarify supports comprehensive FM evaluation with automatic assessment capabilities for generative AI applications, measuring accuracy, robustness, and toxicity to support responsible AI initiatives. For sophisticated content requiring human judgment, organizations can utilize internal workforces or AWS-managed review teams.

Responsible Dataset Preparation

Responsible AI implementation requires carefully prepared, balanced datasets for model training. SageMaker AI Clarify and SageMaker Data Wrangler assist in achieving dataset balance, crucial for creating fair AI models without discriminatory biases.

Inclusive Data Collection

Balanced datasets prevent unfair discrimination and unwanted biases through inclusive, diverse data collection processes that accurately represent required perspectives and experiences. This includes incorporating varied sources, viewpoints, and demographics to ensure unbiased system performance. While particularly critical for human-focused data due to potential societal harm and legal implications, inclusiveness should be prioritized regardless of subject matter.

Dataset Curation

Dataset curation involves labeling, organizing, and preprocessing data for optimal model performance. This process ensures data representativeness while eliminating biases and accuracy-impacting issues. Effective curation guarantees AI models train on high-quality, reliable, task-relevant data through preprocessing, augmentation, and regular auditing procedures.

Model Interpretability and Explainability

Interpretability provides system access enabling human interpretation of model outputs based on weights and features. Explainability involves translating ML model behavior into human-understandable terms. While complex "black box" models resist full comprehension, model-agnostic methods (partial dependence plots, SHAP analysis, surrogate models) reveal meaningful connections between input attributions and outputs, enabling AI/ML model behavior explanation.

Ensuring Model Safety

Model safety encompasses an AI system's ability to avoid causing harm through world interactions, including preventing social harm from biased decision-making algorithms and avoiding privacy/security vulnerabilities. This ensures AI systems benefit society without harming individuals or groups.

Amplified Decision-Making Design

Designing for amplified decision-making helps mitigate critical errors through clarity, simplicity, usability, reflexivity, and accountability principles.

Reinforcement Learning from Human Feedback (RLHF)

RLHF represents an ML technique utilizing human feedback to optimize model self-learning efficiency. While reinforcement learning trains software for reward-maximizing decisions, RLHF incorporates human feedback into reward functions, aligning ML model performance with human objectives, preferences, and requirements across both traditional and generative AI applications.

Conclusion

Implementing responsible AI requires comprehensive attention to multiple interconnected dimensions, from initial dataset preparation through ongoing model monitoring. By leveraging appropriate tools and frameworks while maintaining focus on ethical principles, organizations can develop AI systems that deliver value while minimizing potential harm and maintaining public trust.

Best practices for managing Terraform State files in AWS

Arun Kumar — Tue, 01 Apr 2025 07:02:44 +0000

Customers want to reduce manual operations for deploying and maintaining their infrastructure. The recommended method to deploy and manage infrastructure on AWS is to follow Infrastructure-As-Code (IaC) model using tools like AWS CloudFormation, AWS Cloud Development Kit (AWS CDK) or Terraform.

One of the critical components in terraform is managing the state file which keeps track of your configuration and resources. When you run terraform in an AWS CI/CD pipeline the state file has to be stored in a secured, common path to which the pipeline has access to. You need a mechanism to lock it when multiple developers in the team want to access it at the same time.

By default, the state file is stored locally where terraform runs, which is not a problem if you are a single developer working on the deployment. However if not, it is not ideal to store state files locally as you may run into following problems:

When working in teams or collaborative environments, multiple people need access to the state file
Data in the state file is stored in plain text which may contain secrets or sensitive information
Local files can get lost, corrupted, or deleted.

The recommended practice for managing state files is to use terraform’s built-in support for remote backends. These are:

Remote backend on Amazon Simple Storage Service (Amazon S3): You can configure terraform to store state files in an Amazon S3 bucket which provides a durable and scalable storage solution. Storing on Amazon S3 also enables collaboration that allows you to share state file with others.

Remote backend on Amazon S3 with Amazon DynamoDB: In addition to using an Amazon S3 bucket for managing the files, you can use an Amazon DynamoDB table to lock the state file. This will allow only one person to modify a particular state file at any given time. It will help to avoid conflicts and enable safe concurrent access to the state file.

There are other options available as well such as remote backend on terraform cloud and third party backends. Ultimately, the best method for managing terraform state files on AWS will depend on your specific requirements.

When deploying terraform on AWS, the preferred choice of managing state is using Amazon S3 with Amazon DynamoDB.

For more details on AWS configurations for managing state files, Design Architecture and an example of how efficiently you can manage them in a Continuous Integration pipeline in AWS, check out my blog post on official AWS channel.

DMS Configuration for Oracle to RDS migration

Arun Kumar — Sun, 13 Jun 2021 15:53:01 +0000

Introduction

The following will help you setup Oracle to RDS database migration.

Goals

The following guide will help bootstrap your customers Oracle On Premise → RDS migrations, Oracle RDS → Oracle RDS migrations, etc.

Details

DMS consists of 3 main components

An instance that does all the heavy lifting Endpoints, for target and source databases
A task that describes the loading and replication activities
Tasks are where the bulk of the effort is in tuning the replication and load activities.

There are 3 types of tasks

Full Load
Replication (CDC)
Full load and replication

Source databases are expected to take a 2–5% hit in resource consumption during the full load and sync operations, for a database of any reasonable size expect to be pushing about 20Mbps worth of traffic consistently during replication operations. Full load operations will use considerably higher bandwidth.
For tables with a primary key we will only enable supplemental logging on the column that contains the primary key, for tables with no primary key then we need it enabled it on all columns. If you have particularly busy Oracle schema, this will cause an impact to your performance. Careful of your potential resource contention issues.

Getting Setup

VPC configuration

Don’t use FQDN. Just use the primary IP address of your RDS (Target) and your on premise over Direct Connect Oracle server. AWS have confirmed that custom DNS servers do not work, the resources created by DMS do not inherit custom DNS configuration.

Source Database/Schema Preparation (On Premise)

You need to create a user and grant them some pretty heavy permissions on both the Source and Target database instances.
You can find the AWS provided permissions here; or use the script below to help setup your DMS user on the source Oracle DB.

Outcomes

Correct Permissions for DMS user to replicate data
Enable archive logging if its not already enabled, and explain the archive log destination id
Find the tables with a primary key and list them, then create your alter statement around that list
Find the tables without a primary key, then create your alter statement around that list

If you know there are tables in the schema that don’t need to move, it is a good idea to identify those now.

For execution; I use SQL Developer or SQL Plus depending on what I need to do and my connectivity options.

If you are setting up RDS Oracle to RDS Oracle migrations the Source preparation is slightly different.

-- ON SOURCE DB (NON RDS) (as SYSDBA)
-- DMS onnection attitrubtes addSupplementalLogging=Y;readTableSpaceName=true;archivedLogDestId=1;exposeViews=true
-- other attributes
-- https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.Oracle.html
CREATE user DMS identified by <PASSWORD> default tablespace DATA temporary tablespace DATA_TEMP;
-- Change the above to whatever tablespaces you have configured
Grant CREATE session to DMS;
Grant ALTER ANY TABLE to DMS; 
Grant EXECUTE on dbms_crypto to DMS;
Grant SELECT on ALL_VIEWS to DMS;
Grant SELECT ANY TABLE to DMS;
Grant SELECT ANY TRANSACTION to DMS;
Grant SELECT on V_$ARCHIVED_LOG to DMS;
Grant SELECT on V_$LOG to DMS;
Grant SELECT on V_$LOGFILE to DMS;
Grant SELECT on V_$DATABASE to DMS;
Grant SELECT on V_$THREAD to DMS;
Grant SELECT on V_$PARAMETER to DMS;
Grant SELECT on V_$NLS_PARAMETERS to DMS;
Grant SELECT on V_$TIMEZONE_NAMES to DMS;
Grant SELECT on V_$TRANSACTION to DMS;
Grant SELECT on ALL_INDEXES to DMS;
Grant SELECT on ALL_OBJECTS to DMS;
Grant SELECT on DBA_OBJECTS to DMS; 
Grant SELECT on ALL_TABLES to DMS;
Grant SELECT on ALL_USERS to DMS;
Grant SELECT on ALL_CATALOG to DMS;
Grant SELECT on ALL_CONSTRAINTS to DMS;
Grant SELECT on ALL_CONS_COLUMNS to DMS;
Grant SELECT on ALL_TAB_COLS to DMS;
Grant SELECT on ALL_IND_COLUMNS to DMS;
Grant SELECT on ALL_LOG_GROUPS to DMS;
Grant SELECT on SYS.DBA_REGISTRY to DMS;
Grant SELECT on SYS.OBJ$ to DMS;
Grant SELECT on DBA_TABLESPACES to DMS;
Grant SELECT on ALL_TAB_PARTITIONS to DMS;
Grant SELECT on ALL_ENCRYPTED_COLUMNS to DMS;
Grant SELECT on V_$LOGMNR_LOGS to DMS;
Grant SELECT on V_$LOGMNR_CONTENTS to DMS;
Grant LOGMINING TO DMS;
Grant EXECUTE ON dbms_logmnr TO DMS;
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- Ensure archive logging is enabled and if its not, here is how to enable it (CAUTION. This turns off the database if "shutdown immediate" wasn't clear enough.)

shutdown immediate;
startup mount;
alter database archivelog;
alter database open;
select dest_id,dest_name, status, destination from v$archive_dest;

-- find tables which have a primary key.

select at.TABLE_NAME
from all_tables at
where not exists (select 1
from all_constraints ac
where ac.owner = at.owner
and ac.table_name = at.table_name
and ac.constraint_type = 'P')
and at.owner = '<SCHEMA>';

-- This will list the tables with a primary key, you can then construct your supplemental logging statement based on the column which has the primay key.
-- ALTER TABLE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS

SELECT DISTINCT (a.table_name)
FROM ALL_CONS_COLUMNS A
JOIN ALL_CONSTRAINTS C
ON A.CONSTRAINT_NAME = C.CONSTRAINT_NAME
WHERE
C.CONSTRAINT_TYPE not in('P')
and a.owner ='<SCHEMA>';

-- This will list all the tables without a primary key, these tables need supplemental logging on all columns.
-- If you run the following and then execute all statements generated, ensure that supplemental logging is enabled on ALL TABLES and COLUMNS.
-- Which will cause a greater performance hit. YMMV.
-- select 'ALTER TABLE '||table_name||' ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;' from all_tables where owner = ''

Target Preparation (RDS)

If you have tablespaces with specific block sizes (4k, 8k, 16, etc) then you need to setup your RDS parameter group to allow for caching according to the size of those table spaces. Example below.

ParameterGroup:
      Properties:
        Description: saa-oracle-requirements
        Family: oracle-ee-12.1
        Parameters:
          db_4k_cache_size: '61440'

You can find the required AWS provided permissions here; or use the following to help setup your schema user on the target RDS Oracle DB.

Outcomes:

Correct permissions for user to ingest data
Create some tables spaces

-- ON TARGET RDS DATABASE 
 -- connection attitrubtes n/a
 -- other attributes n/a
 -- Use the name of the schema you want to import below.
 create user SCHEMA identified by <PASSWORD>;
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$ARCHIVED_LOG','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$LOG','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGFILE','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$DATABASE','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$THREAD','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$PARAMETER','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$NLS_PARAMETERS','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$TIMEZONE_NAMES','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$TRANSACTION','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('DBA_REGISTRY','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('OBJ$','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('ALL_ENCRYPTED_COLUMNS','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGMNR_LOGS','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('V_$LOGMNR_CONTENTS','SCHEMA','SELECT');
 exec rdsadmin.rdsadmin_util.grant_sys_object('DBMS_LOGMNR','SCHEMA','EXECUTE');
 exec rdsadmin.rdsadmin_util.grant_sys_object('DBMS_CRYTPO', 'SCHEMA','EXECUTE');
 grant SELECT ANY TRANSACTION to SCHEMA;
 grant SELECT on V$NLS_PARAMETERS to SCHEMA;
 grant SELECT on V$TIMEZONE_NAMES to SCHEMA;
 grant SELECT on ALL_INDEXES to SCHEMA;
 grant SELECT on ALL_OBJECTS to SCHEMA;
 grant SELECT on DBA_OBJECTS to SCHEMA;
 grant SELECT on ALL_TABLES to SCHEMA;
 grant SELECT on ALL_USERS to SCHEMA;
 grant SELECT on ALL_CATALOG to SCHEMA;
 grant SELECT on ALL_CONSTRAINTS to SCHEMA;
 grant SELECT on ALL_CONS_COLUMNS to SCHEMA;
 grant SELECT on ALL_TAB_COLS to SCHEMA;
 grant SELECT on ALL_IND_COLUMNS to SCHEMA;
 grant DROP ANY TABLE to SCHEMA;
 grant SELECT ANY TABLE to SCHEMA;
 grant INSERT ANY TABLE to SCHEMA;
 grant UPDATE ANY TABLE to SCHEMA;
 grant CREATE ANY TABLE to SCHEMA;
 grant CREATE ANY VIEW to SCHEMA;
 grant DROP ANY VIEW to SCHEMA;
 grant CREATE ANY PROCEDURE to SCHEMA;
 grant ALTER ANY PROCEDURE to SCHEMA;
 grant DROP ANY PROCEDURE to SCHEMA;
 grant CREATE ANY SEQUENCE to SCHEMA;
 grant CREATE ANY TABLESPACE to SCHEMA;
 grant CREATE ANY TABLE to SCHEMA;
 grant ALTER ANY SEQUENCE to SCHEMA;
 grant DROP ANY SEQUENCE to SCHEMA
 grant select on DBA_USERS to SCHEMA;
 grant select on DBA_TAB_PRIVS to SCHEMA;
 grant select on DBA_OBJECTS to SCHEMA;
 grant select on DBA_SYNONYMS to SCHEMA;
 grant select on DBA_SEQUENCES to SCHEMA;
 grant select on DBA_TYPES to SCHEMA;
 grant select on DBA_INDEXES to SCHEMA;
 grant select on DBA_TABLES to SCHEMA;
 grant select on DBA_TRIGGERS to SCHEMA;
 grant UNLIMITED TABLESPACE to SCHEMA;
 grant CREATE SESSION to SCHEMA;
 grant DROP ANY TABLE to SCHEMA;
 grant ALTER ANY TABLE to SCHEMA;
 grant CREATE ANY INDEX to SCHEMA;
 grant LOCK ANY TABLE to SCHEMA;
 create tablespace DATA;
 create tablespace DATA_4k blocksize 4K;

Replication Instances

[https://docs.aws.amazon.com/cli/latest/reference/dms/create-replication-instance.html]

Provision a well sized replication instance, recommend “r” class instances due to memory optimization.
If this is for production, make it multi-AZ. Give the instance roughly 150% of the storage for your database.

Endpoint Creation

Source

Remember to use the IP address, not FQDN.
Remember to add the extra connection attributes from the SQL file above.

Target

Again, IP address not FQDN.
No additional connection attributes should be necessary, unless you need to disable useDirectPathFullLoad. See the documentation for why you might want to do that..

[https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.Oracle.html]

Tasks

Right. You are finally ready to start debugging replication. Good luck and godspeed.

Don’t use the AWS Console for creating or modifying tasks. Use the CLI if you’re doing this manually. Half the options are hidden in the console.

I highly recommend that you do this schema by schema, with a separate task for each schema you need to migrate. It will make debugging a lot easier.

Also note, global stored procs and triggers aren’t coming with us. Drop these to SQL and manually create them after you finish the full load operation. Schema level indexes, views, etc should come with us. Table level constraints will as well. If you are having issues with triggers etc, I would recommend that you again drop these to SQL and apply them before the full load, then disable the trigger and enable it after full load is complete. I have included some actions in the debugging section that will help.

What do you want to import?

You’ll need to construct a json file which has the an include statement and ordered list of the schema tables you want to import.

I don’t recommend using a wildcard (%) and then excluding the tables you don’t want, and the reason for this is during debugging if you have a problematic table you can just remove that section from your configuration, drop the table from the target and try again or create a separate task just for that table while you debug the issue.

More information here: [https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html]

ImportTables.json

{
    "rules": [{
            "rule-type": "selection",
            "rule-id": "1", // unique
            "rule-name": "1", // unique 
            "object-locator": {
                "schema-name": "<SCHEMA>",
                "table-name": "TABLE_NAME_ACCEPTS_WILDCARDS_%"
            },
            "rule-action": "include"
        }
}

Task Config
Next you need to configure the task.

There are a few things we want to change from the defaults, otherwise I will leave the investigation of each individual option up to you.

[https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html]

Only truncate tables that exist, so we keep the metadata. We do this if we have to create any tables in advance, because dropping the table will remove metadata and we just want to clear the contents.
Logging — We want detailed debug logs, or at least debug logs. The defaults don’t give you a lot to work with.

TaskConfig.json

{
    "TargetMetadata": {
        "TargetSchema": "<SCHEMA>",      // the schema name we want to import to.
        "SupportLobs": true,
        "FullLobMode": true,
        "LobChunkSize": 64,
        "LimitedSizeLobMode": false,
        "LobMaxSize": 0,
        "LoadMaxFileSize": 0,
        "ParallelLoadThreads": 0,
        "ParallelLoadBufferSize": 0,
        "BatchApplyEnabled": true,
        "TaskRecoveryTableEnabled": true
    },
    "FullLoadSettings": {
        "TargetTablePrepMode": "TRUNCATE_BEFORE_LOAD", // this one. 
        "CreatePkAfterFullLoad": false,
        "StopTaskCachedChangesApplied": false,
        "StopTaskCachedChangesNotApplied": false,
        "MaxFullLoadSubTasks": 8,
        "TransactionConsistencyTimeout": 600,
        "CommitRate": 10000
    },
    "Logging": {                    // This is where we change the severity of the logging information. 
        "EnableLogging": true,
        "LogComponents": [{
                "Id": "SOURCE_UNLOAD",
                "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
            },
            {
                "Id": "TARGET_LOAD",
                "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
            },
            {
                "Id": "SOURCE_CAPTURE",
                "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
            },
            {
                "Id": "TARGET_APPLY",
                "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
            }, {
                "Id": "TASK_MANAGER",
                "Severity": "LOGGER_SEVERITY_DETAILED_DEBUG"
            }
        ],
        "CloudWatchLogGroup": "",
        "CloudWatchLogStream": ""
    },
    "ControlTablesSettings": {
        "historyTimeslotInMinutes": 5,
        "ControlSchema": "",
        "HistoryTimeslotInMinutes": 5,
        "HistoryTableEnabled": true,
        "SuspendedTablesTableEnabled": true,
        "StatusTableEnabled": true
    },
    "StreamBufferSettings": {
        "StreamBufferCount": 3,
        "StreamBufferSizeInMB": 8,
        "CtrlStreamBufferSizeInMB": 5
    },
    "ChangeProcessingDdlHandlingPolicy": {
        "HandleSourceTableDropped": true,
        "HandleSourceTableTruncated": true,
        "HandleSourceTableAltered": true
    },
    "ErrorBehavior": {
        "DataErrorPolicy": "LOG_ERROR",
        "DataTruncationErrorPolicy": "LOG_ERROR",
        "DataErrorEscalationPolicy": "SUSPEND_TABLE",
        "DataErrorEscalationCount": 0,
        "TableErrorPolicy": "SUSPEND_TABLE",
        "TableErrorEscalationPolicy": "STOP_TASK",
        "TableErrorEscalationCount": 0,
        "RecoverableErrorCount": -1,
        "RecoverableErrorInterval": 5,
        "RecoverableErrorThrottling": true,
        "RecoverableErrorThrottlingMax": 1800,
        "ApplyErrorDeletePolicy": "IGNORE_RECORD",
        "ApplyErrorInsertPolicy": "LOG_ERROR",
        "ApplyErrorUpdatePolicy": "LOG_ERROR",
        "ApplyErrorEscalationPolicy": "LOG_ERROR",
        "ApplyErrorEscalationCount": 0,
        "ApplyErrorFailOnTruncationDdl": false,
        "FullLoadIgnoreConflicts": true,
        "FailOnTransactionConsistencyBreached": false,
        "FailOnNoTablesCaptured": false
    },
    "ChangeProcessingTuning": {
        "BatchApplyPreserveTransaction": true,
        "BatchApplyTimeoutMin": 1,
        "BatchApplyTimeoutMax": 30,
        "BatchApplyMemoryLimit": 500,
        "BatchSplitSize": 0,
        "MinTransactionSize": 1000,
        "CommitTimeout": 1,
        "MemoryLimitTotal": 1024,
        "MemoryKeepTime": 60,
        "StatementCacheSize": 50
    },
    "ValidationSettings": {
        "EnableValidation": true,
        "ValidationMode": "ROW_LEVEL",
        "ThreadCount": 5,
        "PartitionSize": 10000,
        "FailureMaxCount": 10000,
        "RecordFailureDelayInMinutes": 5,
        "RecordSuspendDelayInMinutes": 30,
        "MaxKeyColumnSize": 8096,
        "TableFailureMaxCount": 1000,
        "ValidationOnly": false,
        "HandleCollationDiff": false,
        "RecordFailureDelayLimitInMinutes": 0
    }
}

Now you can create and run the task. [https://docs.aws.amazon.com/cli/latest/reference/dms/create-replication-task.html]

aws dms create-replication-task --replication-task-identifier <value> \
 --source-endpoint-arn <value> \ 
 --target-endpoint-arn <value> \ 
 --replication-instance-arn <value> \ 
 --migration-type full-load-and-cdc \ 
 --table-mappings ImportTables.json \
 --replication-task-settings TaskConfig.json

Debugging
If you followed the above initial setup, configuration issues should be minimal.

Cloudwatch logs are enabled for DMS and in the above setup we configured them to be at detailed debug level, this should give you an accurate error message in CloudWatch when a table import misbehaves.

Unfortunately this is where things begin to fall into the realms of DBA territory… but essentially read the logs, weaponise google search, get the DBA guy to help you out. It’s not impossible and in nonproduction you basically get unlimited retries.

Suggestions

Multiple, smaller replication task over 1 all inclusive replication task. This will increase your throughput so be mindful of any resource constraints. However, this makes it easier to debug.
I used 3 tasks, 1 for the majority of tables and 2 smaller jobs for misc tables or tables that were identified as troublesome during our nonproduction DMS migrations. Order your imports.
If you have a table that isn’t importing properly due to a constraint or trigger, disable those on the target during full load and then once that is complete, enable them for CDC replication.
Constraints can be turned on and off.

declare
begin
for c1 in (select y1.table_name, y1.constraint_name from user_constraints y1, user_tables x1 where x1.table_name = y1.table_name order by y1.r_constraint_name nulls last) loop
    begin
        dbms_output.put_line('alter table '||c1.table_name||' disable constraint '||c1.constraint_name || ';');
        execute immediate  ('alter table '||c1.table_name||' disable constraint '||c1.constraint_name);
    end;
end loop;

-- uncomment to truncate the table after disabling the constraint.
-- for t1 in (select table_name from user_tables) loop
--     begin
--         dbms_output.put_line('truncate table '||t1.table_name || ';');   
--         execute immediate ('truncate table '||t1.table_name);
--     end;
-- end loop;

for c2 in (select y2.table_name, y2.constraint_name from user_constraints y2, user_tables x2 where x2.table_name = y2.table_name order by y2.r_constraint_name nulls first) loop
    begin
        dbms_output.put_line('alter table '||c2.table_name||' enable constraint '||c2.constraint_name || ';');       
        execute immediate ('alter table '||c2.table_name||' enable constraint '||c2.constraint_name);
    end;
end loop;
end;

Disable/Enable Triggers

CREATE OR REPLACE PROCEDURE ALTER_ALL_TRIGGERS(status VARCHAR2) IS
  CURSOR c_tr IS (SELECT 'ALTER TRIGGER ' || trigger_name AS stmnt FROM user_triggers);
BEGIN
  IF status NOT IN ('ENABLE', 'enable', 'DISABLE', 'disable') THEN
    DBMS_OUTPUT.PUT_LINE('ONLY ''ENABLEDISABLE'' ACCEPTED AS PARAMETERS');
    RAISE VALUE_ERROR;
  END IF;
  FOR tr IN c_tr LOOP
    EXECUTE IMMEDIATE tr.stmnt || ' ' || status;
  END LOOP;
END;

EXEC ALTER_ALL_TRIGGERS('DISABLE');
EXEC ALTER_ALL_TRIGGERS('ENABLE');

Use the table statics tab to check out the progress of the DMS task. From here you can also drop a table and reload its data. This can be a useful tool for small tables where validation fails and you make a change but don’t want to restart the whole task.
Read the logs. They will often write out the SQL that failed which you can then replay to debug.
Finally if all else is failing, remember that dropping an entire table to SQL and manually importing it isn’t the end of the world (depending of course on number of rows). Just exclude it from your TableImport.json and move on.

Validation

After successful Full load and CDC has kicked off, your tasks will take a status of “Load Complete, replication ongoing”.

AWS DMS runs its own validation to ensure that the tables are aligned, which will throw a status of “TableError” in table statistics and “Error” on Tasks screen.

It is now time to get the Application team and DBA to check the schema and table data. You can also run your own validation steps, write some SQL and do a spot check across both databases. The simplest test is of course, if the application works and passes regression testing on the new database.

Verifying SSL connection for PostgreSQL

Arun Kumar — Sun, 13 Jun 2021 15:14:19 +0000

Steps

Download PGAdmin from [https://www.pgadmin.org/]
After installing, the tool gui will run in your default browser, select Object->Create->Server

Populate the General and Connection tab.

Once the connection is fine, right click “Database=postgres”, select Query Tool as below and execute below SQL.

Docker setup on Windows

Arun Kumar — Sun, 13 Jun 2021 15:09:27 +0000

Steps

a. Install Docker Desktop for Windows.

[https://hub.docker.com/editions/community/docker-ce-desktop-windows]

b. Install WSL2.

[https://docs.microsoft.com/en-us/windows/wsl/install-win10]

c. Make sure awscli is up to date.

[https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html]

d. Run the following command to login to the ECR repo.

aws ecr get-login-password --region <aws-region> | docker login --username AWS --password-stdin <ecr-repo-name>

Or if you are using an older awscli, you can try

aws ecr get-login --no-include-email --region <aws-region> > ./run.sh

Then run the shell file — run.sh

e. If you encounter error in step 4 (d) →

"Error saving credentials: error storing credentials - err: exit status 1"

then you need to rename the following exe file.

And also rename ~/.docker/config.json to ~/.docker/config.json.original

Run the shell script run.sh again.

Copy AWS ECR repo between 2 AWS accounts

a. To pull a repo:

docker pull <repo/image>

b. Create the same repo in second account.

aws ecr create-repository — repository-name <repo-name> --profile account2

c. Tag the image and push to target repo.

docker tag <account1-image> <account2-image>

docker push <account2-image>

Spot Instance Scenarios

Arun Kumar — Sun, 13 Jun 2021 14:55:45 +0000

Scenario

Instance stopped by AWS due to Insufficient Capacity but not started automatically by AWS when Capacity is available again.
No issue when user start the instance manually.

Reason

Service role is not able to access the KMS key that is cross account and the instance is using this KMS for their volume.

Troubleshooting

Look at the configuration changes and you will see “Client error on launch”.

Check CloudTrail logs and you can see the Access Denied error on KMS.

Solution

On the account whereby the instance is launched, run the following command to grant the KMS permission to service role.

aws kms create-grant — region <region> –key-id <arn of the KMS> — grantee-principal <arn of the Spot Service Role> — operations “Decrypt” “Encrypt” “GenerateDataKey” “GenerateDataKeyWithoutPlaintext” “CreateGrant” “DescribeKey” “ReEncryptFrom” “ReEncryptTo”

Example:

aws kms create-grant — region ap-southeast-1 — key-id arn:aws:kms:ap-southeast-1:123456789:key/479d6414-e442–4873–9b10-123dwdas343 — grantee-principal arn:aws:iam::987654321:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot — operations “Decrypt” “Encrypt” “GenerateDataKey” “GenerateDataKeyWithoutPlaintext” “CreateGrant” “DescribeKey” “ReEncryptFrom” “ReEncryptTo”

Result:

Note:
Monitor the situation to ensure that instance starts up whenever spot instance is reclaimed by AWS due to insufficient capacity.

How to increase/decrease FileSystem under LVM

Arun Kumar — Sun, 13 Jun 2021 14:00:17 +0000

Steps

a. Before starting, ensure the following packages are installed

e2fsprogs
lvm2

b. Run pvdisplay to check which EBS is under LVM.

sudo pvdisplay

c. Run vgdisplay to check unallocated space available for a volume group

sudo vgdisplay

d. Find the lvm fs that you want to increase/decrease.

sudo lvscan

e. Example: Increase vol17 size from 5Gb to 15Gb

>>lvextend -L15G /dev/vgs1/vol17 --resizefs

f. Example: Reduce vol17 size from 5Gb to 1Gb ( you can only do this if no process is holding to the fs. Best is to unmount the fs)

Important to ensure package e2fsprogs is installed, else the below command can cause superblock/partition table corruption.

>>lvreduce -L1G /dev/vgs1/vol17 --resizefs

g. If the EBS is resized, then need to do the following, example /dev/xvdg was resize from 950Gb to 1150Gb

lsblk
> growpart /dev/xvdg 1

h. Resize the physical volumes using the pvresize command.

pvresize /dev/xvdg1

i. Use vgdisplay to see the increased free size. To see what fs type that the LVM fs is run,

lsblk -f

j.
i. If you are restoring an EBS snapshot with LVM, after mounting the restored EBS volume, then you need to do the below.

sudo yum install lvm2 (if lvm command not found )
sudo vgchange -ay

ii. If doing pvdisplay shows an unknown physical volume, then run command

vgreduce — removemissing <VG NAME>

k. To increase swap space under LVM, do the following

>>swapoff -v /dev/vgs1/vol54swap

Disable the vol that contain the swap, might need to stop appls if swap is in-use and free memory not available.

>>lvextend -L32G /dev/vgs1/vol54swap
>>mkswap /dev/vgs1/vol54swap
>>swapon -va
>>cat /proc/swaps

l. If the disk increased is greater than 2Tb and is using MBR partition type, you need to convert to GPT partition type.

gdisk /dev/nvme1n1

[https://superuser.com/questions/1250895/converting-between-gpt-and-mbr-hard-drive-without-losing-data]

How to Backup EFS using AWS Backup

Arun Kumar — Sun, 13 Jun 2021 12:05:34 +0000

Backup Creation for EFS

AWS Backup now supports Single File Restore for EFS.
You can create backup by creating a backup plan, assigning it to EFS and then deploy it.

Refer below steps

Restoring Backup for EFS

EFS file system restoration can be “Full restore” or through “Item-level restore” of the file system.

Full restore — it restores the filesystem in its entirety including all root level folders and files.
Item-level restore — you can select and restore up to 5 items within your Elastic File System. Enter a relative path to a file or folder.

EFS Backup can be restored to the new EFS file system or to the source EFS file system using AWS Backup API, CLI, or AWS Console.

AWS Backup (Console)

i. Open the AWS Backup console.

ii. In the navigation pane, choose Protected resources. A list of your recovery points, including the resource type, is displayed by Resource Id. Choose a resource to open the Backups pane.

iii. To restore a resource, choose the radio button next to the recovery point in the Backups pane, and then choose Restore in the upper-right corner of the pane.

iv. Specify the restore parameters. Select restoration method by either:

a. Restore to a new file system.

b. Restore to directory in source file system.

v. For IAM role, choose Default role.

Note: If the AWS Backup default role is not present in your account, one will be created for you with the correct permissions.

vi. Choose Restore resource.

You are brought to the restore jobs table, and you should see a message at the top of the page informing you about the restore job.
The message also includes a link to the service console of the resource that you just restored.
You can switch to that console and take action on the new resource that you created from the backup.

AWS Backup (CLI)

Amazon EFS Restore Metadata

Use the following information to restore an Amazon Elastic File System (Amazon EFS) instance:

file-system-id — ID of the Amazon EFS file system that is backed up by AWS Backup. Returned in GetRecoveryPointRestoreMetadata.

Encrypted — A Boolean value that, if true, specifies that the file system is encrypted. If KmsKeyId is specified, Encrypted must be set to true.

KmsKeyId — Specifies the AWS KMS key that is used to encrypt the restored file system.

PerformanceMode — Specifies the throughput mode of the file system.

CreationToken — A user-supplied value that ensures the uniqueness (idempotency) of the request.

newFileSystem — A Boolean value that, if true, specifies that the recovery point is restored to a new Amazon EFS file system.

Example:

aws backup start-restore-job --recovery-point-arn arn:aws:backup:ap-southeast-1:123456789:recovery-point:f95aab35-b90a-4e40-8269-b43797c5234df5 --metadata file-system-id=fs-2de30e6c,Encrypted=true,PerformanceMode=generalPurpose,newFileSystem=true,KmsKeyId=aws/elasticfilesystem,CreationToken=efsrestore --iam-role-arn arn:aws:iam::123456789/service-role/AWSBackupDefaultServiceRole --resource-type EFS

Reference

[https://docs.aws.amazon.com/cli/latest/reference/backup/start-restore-job.html]
[https://docs.aws.amazon.com/aws-backup/latest/devguide/API_StartRestoreJob.html]

Cross account role access to S3 in another AWS account

Arun Kumar — Sun, 13 Jun 2021 11:56:55 +0000

Scenario

Need to access S3 in a different AWS account from EC2 in your account.

Steps

For the EC2 role on the first AWS account, add the following in-line policy. (For the KMS key, make sure it is the one created for the same one as the target s3 bucket)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:List*",
                "s3:Put*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name",
                "arn:aws:s3:::bucket-name/*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": [
                "arn:aws:kms:ap-southeast-1:123456789:key/123ddwq-123d-123fd34-553f"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant",
                "kms:RetireGrant",
                "kms:ListRetirableGrants"
            ],
            "Resource": [
                "arn:aws:kms:ap-southeast-1:987654321:key/3136e26c-3144-12fd-432r4-34rf4244f"
            ],
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            },
            "Effect": "Allow"
        }
    ]
}

On the Second AWS Account, IAM → Encryption Keys → Customer managed key, add the EC2 Account to allow access to S3.
Update the S3 bucket policy. Example below.

{
            "Sid": "Stmt1357935647218",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::1234556789:root"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::bucket-name"
},
{
            "Sid": "Stmt1357935648634",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:root"
            },
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "arn:aws:s3:::bucket-name/*"
}

Test and verify the access !

S3 Same-Region Replication (SRR) vs Cross-Region Replication (CRR)

Arun Kumar — Sun, 13 Jun 2021 02:34:07 +0000

Overview

This document is to evaluate and provide guidance on the benefits/feature of Amazon S3’s SRR and CRR replication strategy.
Replication enables automatic, asynchronous copying of objects across Amazon S3 buckets. Buckets that are configured for object replication can be owned by the same AWS account or by different accounts.
Replication can be achieved within the same AWS Region or different AWS Region.

Use Cases

Same-Region Replication (SRR)

Automatically replicates data between buckets within the same AWS Region.
Replication can be setup at a bucket level, a shared prefix level, or an object level using S3 object tags.
SRR can be use to make a second copy of data in the same AWS Region.
Helps to address data sovereignty and compliance requirements by keeping a copy of your data in a separate AWS account in the same region as the original.
Allows to change account ownership for the replicated objects to protect data from accidental deletion.
Allows to aggregate logs from different S3 buckets for in-region processing, or to configure live replication between test and development environment.

Observations

Both source and target buckets must be version enabled.
Object deletions are not replicated to target bucket (so it’s not like rsync — delete).

Cross-Region Replication (CRR)

Automatically replicates data between buckets across different AWS Regions.
Provides ability to replicate data at a bucket level, a shared prefix level, or an object level using S3 object tags.
CRR provide lower-latency data access in different geographic regions.
CRR can help with compliance requirement to store copies of data hundreds of miles apart.
Allows to change account ownership for the replicated objects to protect data from accidental deletion.

Security

Objects are remained encrypted throughout the replication process.
The encrypted objects are transmitted securely via SSL within the same region (if using SRR) or from the source region to the destination region (if using CRR).

Pricing for S3 Replication

For CRR and SRR, Amazon S3 charges for storage in the selected destination S3 storage class, in addition to the storage charges for the primary copy, and replication PUT requests.
For CRR, you will be charge for inter-region Data Transfer OUT from Amazon S3 to your destination region.
Pricing for the replicated copy of storage is based on the destination AWS Region, while pricing for requests and inter-region data transfers are based on the source AWS Region.

Orphaned CloudFormation Stacks — HouseKeeping

Arun Kumar — Fri, 11 Jun 2021 15:57:21 +0000

Scenario

There could be some stacks missed out during teardown process due to some issues and this might leave those stacks orphaned.
Also when App teams create a new stack without deleting their previous stack, this will leave the previous stack orphaned.

Solution

List out all the stacks based on the state in the corresponding account using below python script. Filter the suspecting orphaned stacks from the list.

# Function: EvaluateOrphanedStacks
# Purpose: List out stacks based on the state and accounts
import boto3
import json
from datetime import datetime
from datetime import date
cfn_client=boto3.client('cloudformation')
def list_stacks():
paginator = cfn_client.get_paginator('list_stacks')
response_iterator = paginator.paginate(
StackStatusFilter=[
'CREATE_IN_PROGRESS',
'CREATE_FAILED',
'CREATE_COMPLETE',
'ROLLBACK_IN_PROGRESS',
'ROLLBACK_FAILED',
'ROLLBACK_COMPLETE',
'DELETE_IN_PROGRESS',
'DELETE_FAILED',
'UPDATE_IN_PROGRESS',
'UPDATE_COMPLETE_CLEANUP_IN_PROGRESS',
'UPDATE_COMPLETE',
'UPDATE_ROLLBACK_IN_PROGRESS',
'UPDATE_ROLLBACK_FAILED',
'UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS',
'UPDATE_ROLLBACK_COMPLETE',
'REVIEW_IN_PROGRESS',
'IMPORT_IN_PROGRESS',
'IMPORT_COMPLETE',
'IMPORT_ROLLBACK_IN_PROGRESS',
'IMPORT_ROLLBACK_FAILED',
'IMPORT_ROLLBACK_COMPLETE'
]
)
for page in response_iterator:
for stack in page['StackSummaries']:
print(stack['StackName'])
if __name__ == '__main__':
list_stacks()

Note:
Its ALWAYS recommended and good practice to reduce the orphaned stacks and unwanted resources !

How to recover an EC2 instance that doesn’t boot up

Arun Kumar — Fri, 11 Jun 2021 15:55:10 +0000

Background

Often, you might face issues with your EC2 and in some cases it doesn’t even boot up for you to check and troubleshoot the error. To know more about the error “EC2 instance unable to boot up properly”, check

EC2 -> Action -> Instance Setting -> Get System Log

Steps

Traditional way

Stop the instance
Unmount the root volume and attach the root volume to another instance running the same AMI.

[root@instance-1]:/
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 10G 0 disk
â”œâ”€xvda1 202:1 0 1M 0 part
â””â”€xvda2 202:2 0 10G 0 part /
xvdf 202:80 0 50G 0 disk /prod/applc/wls
xvdg 202:96 0 20G 0 disk /prod/applc/logs
xvdh 202:112 0 10G 0 disk
â”œâ”€xvdh1 202:113 0 1M 0 part
â””â”€xvdh2 202:114 0 10G 0 part

In example above, root is /dev/xvdh

mount -o nouuid /dev/xvdh2 /mnt

Check if the /mnt eg disk space full, or see /mnt/var/log/dmesg
Check if /mnt/etc/fstab got any issue with mount point.
If you don’t put the “nouuid” option, the mount will fail if the volume had the same UUID as the current root volume.
After fixing the issue, detach the Volume and attach back.
When attached back as root, make sure to mount as /dev/xvda or /dev/sda1

Snapshot method

If the above steps still don’t work, then you need to restore from SNAPSHOT. Do the following —

EC2→snapshot. Select the root snapshot and then Action → Create Image.

Change Virtualization type to Hardware-assisted.
Add the snapshot data volume, ensure the dev/sdx match the current instance and the size.
This will create a new AMI image which you can then use to spin up a new EC2 instance.
Once you test and verify this new EC2 instance is fine, you can stop the new instance, take the final backup before you terminate the instance.
If need to reuse the same IP then respin up the new AMI image with the IP required.