Powershell, 7-Zip, Amazon S3 Upload Script with AES-256 Encryption

I recently was tasked with finding a way to store some backup files from our server in a secure and reliable off-site location. After talking with our hosting provider, who wanted around $600 a month for off-site tape rotation, we decided to look at using Amazon Simple Storage Service (Amazon S3) to store the files in the cloud instead. We needed a way to automate the upload process and make sure that the data was encrypted, so I spent a few days working on a Powershell script (using the excellent PowerGui Script Editor) that uses 7-zip to create a .7z archive with AES-256 encryption and then send it up to Amazon S3 using the Amazon Web Services SDK for .NET. Here is the script:

#Description: Powershell 2.0 script to zip and encrypt a folder using 7zip and then send it to Amazon S3 #Author: Greg Bray (2/19/2010) #Website: http://codeblog.theg2.net/2010/02/powershell-7-zip-amazon-s3-upload.html #License: free for personal or commercial use under the Creative Commons Attribution 3.0 United States License. #Prerequisite: You must first install the latest version of 7Zip from http://www.7-zip.org/ (licensed under the GNU LGPL) #Prerequisite: You must also install the Amazon Web Services SDK for .NET from http://aws.amazon.com/sdkfornet/ (licensed under the Apache License, Version 2.0) #Recommended: I recommend using CloudBerry S3 Explorer for managing Amazon S3 files. http://cloudberrylab.com/default.aspx?page=cloudberry-explorer-amazon-s3 #Recommended: I recommend using http://PowerGui.org Script editor if you want to change or debug this script. #Usage: ps7ZiptoAmazonS3 'BackupFolderPath' 'S3ObjectKey' 'TranscriptLogFile' # BackupFolderPath = Folder to backup (do not include a trailing slash) # S3ObjectKey = Object key where to store file on Amazon S3. Note: If object already exists on S3 it will be overwritten. # TranscriptLogFile = Optional log file where the transcript will be appended. #NOTE: you also will need to set the Zip and Encrypt settings, Amazon S3 settings, and 7Zip/AWSSDK file paths below <#Example calling from another powershell script (with log file): & 'c:\Backups\ps7ZiptoAmazonS3.ps1' 'c:\Backups\DataFolder' 'S3Backups/DataFolder.7z' 'c:\Backups\BackupLog.txt' #> <#Example calling from Run or task Scheduler: NOTE: may need to run "Set-ExecutionPolicy RemoteSigned" in powershell to allow unsigned local scripts to execute powershell.exe -Command "& 'c:\Backups\ps7ZiptoAmazonS3.ps1' 'c:\Backups\DataFolder' 'S3Backups/DataFolder.7z' 'c:\Backups\BackupLog.txt'" #> #Check arguments and set variables if ($args.count -ge 2) { $FolderPath = $args[0] #Local folder to encrypt and send to S3 (No trailing slash) $S3ObjectKey = $args[1] #Destination on S3 where file will be stored. Note: If object exists it will be overwritten. if ($args.count -ge 3) { #start logging transcript Start-Transcript $args[2] -Append | Out-Null } } else { #Default preset arguments used if no values are provided $FolderPath = "c:\Backups\DataFolder" $S3ObjectKey = "S3Backups/DataFolder.7z" } write-host "`r`n`r`n" #Start log file with 2 blank lines $dtStart = [System.DateTime]::Now #Save start time to calculate total time below write-host ($dtStart.ToString() + " - Script started") #Zip and Encrypt settings $EncryptionKey = "secret_password" #Data is encrypted using Strong AES-256 encryption. Recommended password size is 10-16 characters. $TemporaryPath = "c:\Temp" #No trailing slash. Used to store zip file localy before uploading. File will be deleted after upload but not if there is an exception (see below). #Amazon S3 account settings (See http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key): $S3AccessKeyID = "ABCDEFGHIJKLMNOPQRST" #Your Amazon S3 Access Ky ID $S3SecretKeyID = "0123456789abcdefghijklmnopqrstuvwzyz" #Your Amazon S3 Secret Access Key $S3BucketName = "S3BackupBucket" #S3 Bucket name where the file will be stored Write-Host ($dtStart.ToString() + " - FolderPath:'$FolderPath' S3BucketName:'$S3BucketName' S3ObjectKey:'$S3ObjectKey'") #Load AWS Assembly and Initialize 7Zip path (only runs once when debugging script) if($7ZipPath -eq $null){ $7ZipPath = "C:\Program Files\7-Zip\7z.exe" #Path to 7Zip $AWSDOTNETSDKPath = "C:\Program Files (x86)\AWS SDK for .NET\bin\AWSSDK.dll" #Path to Amazon SDK for x64 system. Remove " (x86)" when running on an x86 machine if (-not [System.IO.File]::Exists($AWSDOTNETSDKPath)) { #Try path for x86 system $AWSDOTNETSDKPath = $AWSDOTNETSDKPath.Replace(" (x86)","") if (-not [System.IO.File]::Exists($AWSDOTNETSDKPath)) { #File not found, throw exception write-error "ERROR: Cannot find Amazon Web Services SDK. Please download .NET SDK from http://aws.amazon.com/sdkfornet/ and update AWSDOTNETSDKPath in script." exit 2 } } #Load AWS Assembly [Reflection.Assembly]::LoadFile($AWSDOTNETSDKPath) | Out-Null } #Function for computing Amazon S3 MD5 digest (128 bit base64 encoded) using System.Security.Cryptography.MD5 function Hash-MD5 ($file) { $cryMD5 = [System.Security.Cryptography.MD5]::Create() $fStream = New-Object System.IO.StreamReader ($file) $bytHash = $cryMD5.ComputeHash($fStream.BaseStream) $fStream.Close() return [Convert]::ToBase64String($bytHash) } #Zip and Encrypt files Write-Host ([System.DateTime]::Now.ToString() + " - Zip and Encrypt Files") $FolderName = $FolderPath.Substring($FolderPath.LastIndexOf("\")+1) #grab the name of the backup folder $TimeStamp = [datetime]::Now.ToString("yyyy-MM-dd_HHmm") #Create unique timestamp string $strFile = [String]::Format("{0}\{1}_{2}.7z", $TemporaryPath,$FolderName,$TimeStamp) #Create filename for the zip #7z options: add, type 7z, Archive filename, Files to add (with wildcard. Change \* to \prefix* or \*.txt to limit files), compression level 9, password, encrypt filenames, Send output to host so it can be logged. #See http://dotnetperls.com/7-zip-examples or the 7zip help file for more info on command line switches. #NOTE: -mx7 and -mx9 get around 10 to 1 compression for text files and take round 5 minutes for 1GB of data. Use -mx3 or -mx5 to increase speed but decrease compression level. & $7ZipPath "a" "-t7z" "$strFile" "$FolderPath\*" "-mx9" "-r" "-p$EncryptionKey" "-mhe" | out-host #Call 7z.exe to create archive file if($LASTEXITCODE -ne 0){ #Detect errors from 7Zip. Note: 7z will crash sometimes if file already exists Write-Error "ERROR: 7Zip terminated with exit code $LASTEXITCODE. Script halted." exit $LASTEXITCODE } #Create Amazon PutObjectRequest. Details at http://docs.amazonwebservices.com/sdkfornet/latest/apidocs/html/T_Amazon_S3_Model_PutObjectRequest.htm Write-Host ([System.DateTime]::Now.ToString()) " - Creating S3 PutObjectRequest" $AmazonS3 = [Amazon.AWSClientFactory]::CreateAmazonS3Client($S3AccessKeyID, $S3SecretKeyID) $S3PutRequest = New-Object Amazon.S3.Model.PutObjectRequest $S3PutRequest.BucketName = $S3BucketName $S3PutRequest.Key = $S3ObjectKey $S3PutRequest.FilePath = $strFile Write-Host ([System.DateTime]::Now.ToString()) " - Compute File MD5 Digest" $strMD5 = Hash-MD5($strFile) #NOTE: may take 5-30 seconds to compute digest for 1GB file. $S3PutRequest.MD5Digest = $strMD5 #Use MD5 digest to ensure data is not corrupted during transport. $dFileSize = (Get-Item $strFile).Length / 1MB #Get archive size in MB Write-Host ([System.DateTime]::Now.ToString()) ([String]::Format(" - Uploading File:'{0}' Size:{1:#,###.#} MB", $strFile, $dFileSize)) $S3Response = $AmazonS3.PutObject($S3PutRequest) #NOTE: upload defaults to 20 minute timeout. #If upload fails it will throw an exception and $S3Response will be $null if($S3Response -eq $null){ Write-Error "ERROR: Amazon S3 put requrest failed. Script halted." exit 1 } Write-Host ([System.DateTime]::Now.ToString()) " - Delete Local 7z File:'$strFile'" Remove-Item $strFile -Force $dtEnd = [datetime]::Now $tsTotal = $dtEnd.Subtract($dtStart) Write-Host ($dtEnd.ToString()) " - Script finished. Total time: " $tsTotal write-host "`r`n`r`n" #End log file with 2 blank lines if ($args.count -ge 3) { #stop transcript Stop-Transcript }

It seems to work pretty well so far, taking about 5-10 minutes to zip and encrypt 1GB of SQL Backups down to 100MB and then upload it to Amazon S3. From there we can use tools like CloudBerry S3 Explorer to browse or download files when needed. The monthly costs to keep data on Amazon S3 is $0.150 per GB, with $0.10 per GB transfer in and $0.150 per GB transfer out. With a 1 week backup retention and minimal data-out transfers we expect to pay around around $10 to $20 a month and should be able to access it much quicker than if we were using off-site tape storage. Cloud computing FTW!

UPDATE 6/14/2010: I just posted the script used to create the Database backup folder.