In many enterprise environments, exchanging files with external partners is still a partially manual process.
Teams upload files to shared folders, transfer them to external SFTP servers, download incoming files, and manually track whether transfers have completed successfully.
This creates several operational challenges:
- Manual file transfers and repetitive activities.
- Duplicate file uploads.
- Lack of visibility into file-transfer status.
- Security risks associated with files received from external systems.
- Uncontrolled accumulation of files on shared storage and SFTP servers.
- Difficulty identifying which user uploaded a particular file.
- Limited visibility into failed transfers.
I recently worked on building an automated file exchange solution to address these challenges.
For this article, I am calling the solution File Exchange Automation. All project names, folder names, endpoints, and infrastructure identifiers have been generalized.
The solution uses Python, Paramiko, AWS S3, Amazon GuardDuty Malware Protection, Linux EC2, Windows SMB shares, and PowerShell-based auditing.
Let's walk through the architecture and implementation.
1. Architecture Overview
The solution supports bidirectional file transfers between an enterprise shared drive and an external partner's SFTP server.
Outbound workflow:
Enterprise Shared Drive → Linux EC2 → External SFTP Server
Inbound workflow:
External SFTP Server → Linux EC2 → Shared Drive Stage → Amazon S3 → GuardDuty Malware Scanning → Clean/Quarantine → Shared Drive
The automation is built using Python and runs through scheduled jobs on Linux EC2.
2. Technology Stack
Technology:
- Python and Paramiko
- Linux EC2
- Windows SMB Share
- Amazon S3
- Amazon GuardDuty
- AWS Secrets Manager
- IAM Roles
- PowerShell
- Cron
3. How the Automation Works
Outbound file transfer
Users place files in the shared drive's outbound folder. Python scans the directory, connects to the external SFTP server, checks for duplicate filenames, and uploads files that are not already present.
After a successful upload and verification, files are moved to a local archive.
Inbound file transfer
The automation downloads files from the external SFTP server into a staging folder. These files are then uploaded to Amazon S3 for malware scanning.
Security scanning
Amazon GuardDuty Malware Protection scans incoming S3 objects.
NO_THREATS_FOUND: File proceeds to the clean workflow.
THREATS_FOUND: File is isolated in quarantine.
UNSUPPORTED: File requires separate investigation and is not treated as clean.
Approved files are delivered to the shared drive's inbound folder.
4. Additional Automation and Security Features
Beyond basic file transfers, we implemented several operational features:
Automated retention: A scheduled cleanup process removes eligible files from the remote SFTP directory after a configured two-hour retention period.
Capacity management: A combined storage threshold monitors remote SFTP folders and pending outbound files to prevent uncontrolled transfers.
User-visible logging: Daily CSV reports provide file names, timestamps, transfer direction, processing status, and error details. Users can access these logs directly from a shared-drive folder.
File activity auditing: Windows Security Event ID 4663 and a PowerShell collector capture relevant file activity, including Active Directory username, filename, timestamp, and action.
Secure credentials: SFTP credentials are retrieved from AWS Secrets Manager, while IAM roles provide AWS access to the EC2 instance.
5. Challenges and Lessons Learned
During implementation, we encountered several real-world issues:
Large files caused capacity constraints and prolonged synchronization jobs.
SFTP directory listing worked, but remote file creation sometimes failed because of server-side restrictions.
SMB mount inconsistencies caused occasional file access and deletion errors.
Cross-filesystem archive operations required additional file verification and safer handling.
Unsupported malware scan results required separate processing rather than being treated as clean.
Users needed better visibility into transfer progress and failures, leading to the shared CSV logging solution.
These challenges highlighted an important point: successful file-transfer automation requires more than connecting to an SFTP server. It also requires proper error handling, capacity management, security controls, and operational visibility.
6. Final Thoughts
With this solution, we brought file transfer, malware scanning, retention management, and file activity auditing into one automated workflow.
The objective was not just to automate uploads and downloads, but to build a more secure, manageable, and traceable file exchange process.
I'm continuing to improve the automation, particularly around large-file handling, transfer reliability, and monitoring.
How are you handling secure file exchanges in your enterprise environments? Are you using custom Python automation, managed SFTP services, or cloud-native solutions?
I'd love to hear your experiences.
Technologies: Python | AWS | Amazon S3 | Amazon GuardDuty | Paramiko | Linux | PowerShell | DevOps
Top comments (0)