Organize Your Modules

Separating modules by resource type is a good option. Break down your infrastructure into logical components e.g. networking, compute and storage. Always make use of a standard directory structure that might look like this:

/modules
  /networking
  /compute
  /storage
/environments
  /dev
  /staging
  /prod

You should also consider using Git repositories to store and version your modules. For complex environments, consider maintaining a separate repository for shared modules.

Adopt Versioning Practices

Tag your module versions using Semantic Versioning (e.g., v1.0.0) and use these tags in your Terraform configuration. It is always important to lock module versions, specifying a version for each module in your terraform configuration. These two techniques will help in rolling out updates safely and ensuring optimal deployment with no breaking changes.

module "networking" {
  source = "git::https://github.com/example/networking.git?ref=v1.2.0"
}

Use Remote State Management

Store your Terraform state files in Azure Storage Accounts or another remote backend to ensure consistency and enable team collaboration and configure state locking to prevent concurrent modifications. Terraform state will become a critical component of your infrastructure. Without it, you will not be able to perform infrastructure deployments. Always implement resiliency for your Storage Accounts that hold your Terraform state such as Geo-Redundancy, soft-delete for blobs / containers as well as Azure resource locks.

terraform {
  backend "azurerm" {
    resource_group_name = "tfstate-rg"
    storage_account_name = "tfstate"
    container_name = "state"
    key = "terraform.tfstate"
  }
}

Enforce Standards with Module Registry

Use a private Terraform module registry (like Terraform Cloud or an internal solution) to centralize your modules and establish naming conventions and documentation standards for modules to maintain clarity.

Implement CI/CD for Modules

Automate testing and validation of modules using CI/CD pipelines. There are several tools available like terratest or checkov for validating your infrastructure code. In your CI pipeline make sure to include the following steps:

• Lint and format your code.
• Run unit tests and static analysis.
• Deploy to a test environment and validate outputs.

Document Your Modules

Create README.md files for each module with clear usage instructions, input/output variables, and examples. Use the following example as template for your module documentation:

# Networking Module
## Usage
```hcl
module "networking" {
  source = "git::https://github.com/example/networking.git"
  vnet_name = "my-vnet"
  subnet_count = 2
}

Inputs
vnet_name: (string) Name of the Virtual Network
subnet_count: (int) Number of subnets to create
Outputs
subnet_ids: List of subnet IDs

Monitor and Review

Regularly review and update modules to incorporate best practices, security updates, and new Azure features. Use tools like Azure Policy to ensure deployed resources comply with your organization's governance policies.

Foster Collaboration

Encourage team contributions to module repositories and always use pull requests and code reviews to maintain quality and consistency.

By following these practices, you can streamline the management of your Azure Terraform modules, reduce technical debt, and empower your team to deliver robust infrastructure as code. Start small, iterate, and adapt these tips to fit your organization's unique needs.

Recently, I was working on an Azure project and one of my tasks involved a step where I had to upload some files to a Linux virtual machine. However, the environment was locked down and I did not have direct SSH access to the VM. The only option for remote access was this cool service called "Azure Bastion". You probably already know what it does but in case you don't, you can read more about this fully managed RDP/SSH solution.

So, during my research what would be the quickest approach to perform this upload, I first thought that I could upload the file to a remote cloud storage e.g. OneDrive or even an Azure Storage Account. But, I didn't want to involve more services and at some point this post flashed into my mind which says that Azure Bastion supports file uploads with its "Native Client Support" feature.

I though "Great, I already use Bastion!" What about the prerequisites? Easy stuff:

• Azure CLI >= 2.32
• The VM resource ID

Keep reading to find out how you can try this by yourself.

What you will need is of course a Bastion, a Linux VM and some Virtual Networks. If you have none of these, go to Azure Quickstart Templates and deploy this template by clicking on "Deploy to Azure". Sign in to your Azure Subscription and fill in the template parameters. You will need to define at least the Resource Group along with a username and a password for your VM. Once completed, you should have a Resource Group with the following resources:

• A hub vnet
• A spoke vnet
• A bastion host on the Basic tier (you will later need to upgrade to Standard)
• A public IP address
• A Linux VM with its disk and network interface

For the file upload feature to work, your Bastion host must be on the Standard tier and the "Native Client Support" option must be enabled. To do this on the Azure Portal, navigate to your Bastion host resource and under "Settings" select the "Configuration" option. In this page, ensure that the "Standard" tier is selected and the "Native Client Support" box is checked.

Click "Apply" to deploy any changes that you made.

Open a Terminal window and run the following commands replacing <subscription id> by the ID of the subscription that contains the Bastion host (not the VM):

az login
az account set -s <subscription id>

Run the following command to create a tunnel between your PC and the VM over the Bastion host, providing the right parameter values for your environment:

az network bastion tunnel --name <bastion name> --resource-group <resource group name> --target-resource-id <VM resourceId> --resource-port <target VM port> --port <local machine port>

• bastion name: the name of your Bastion host
• resource Group name: the name of the resource group that contains your Bastion host
• VM resourceId: you can find it in the properties section of your VM in the Azure Portal
• Target VM port: it is the port where SSH runs on the targer VM, typically 22.
• Local machine port: choose a non-reserved TCP port on your PC. You will use it later for the file transfer.

Tip: In case you receive a message that the bastion host was not found, make sure that you are targeting the right Azure Subscription in your Azure CLI session. More specifically, if you are on an environment that uses Azure Landing Zones, most probably your bastion host is deployed in the "Connectivity" subscription but your VM is on an Application Landing Zone. Furthermore, to use the Bastion host, you need to have the "Reader" role on the Bastion host as well as the Virtual Network where it lives.

Once the tunnel is successfully created you will see the message below:

Opening tunnel on port: 59999
Tunnel is ready, connect on port 59999
Ctrl + C to close

Now you can perform the file upload using your favourite SFTP/SCP client. I use WinSCP. To connect to your VM use the following host details:

Host: localhost
Port: The local machine port that you provided earlier in the 'az network bastion tunnel' command
Username: The VM username
Password: The VM password

All set! Now you are ready to perform the file upload:

The good news is that you can easily benefit from this and start applying all these best practices on your own Azure environment today! Microsoft provides the following three implementation options for Azure Landing Zones:

• Portal Accelerator (GUI-driven)
• Bicep module (automation-friendly)
• Terraform module (automation-friendly)

As you might have guessed, this blog and its author love automation, therefore we will not focus on bullet #1 (although you can read about it at the Deploy Enterprise-Scale foundation page on GitHub). So, we are left with options #2 and #3. For this post we will work with the Terraform option.

Before we dig deeper, let's make sure that you are familiar with the concepts of Terraform. Terraform is one of the most popular Infrastructure-as-Code languages, that can help you deploy and configure your infrastructure using declarative code syntax. It can work with all major public cloud providers as well as many on-premises platforms. There is a nice tutorial at Hashicorp Learn that can teach you all the basics for Terraform on Azure in less than an hour.

Back to our main topic, Microsoft provides the Azure Landing Zones Enterprise Scale Terraform module, which can help us quickly deploy the foundations of our Azure environment, such as management groups, policies, role assignments, networking, logging and security. To start, make sure that your workstation has all the necessary tools installed such as:

• Your favorite editor (e.g. VSCode)
• Azure CLI
• Terraform

Launch VSCode and create a new folder e.g. "ALZ". Next, create a new file "main.tf." and type the following code:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.0.2"
    }
  }
}

provider "azurerm" {

  features {}

}

data "azurerm_client_config" "core" {}

module "enterprise_scale" {

  source  = "Azure/caf-enterprise-scale/azurerm"

  version = "2.4.1"

providers = {

    azurerm              = azurerm

    azurerm.connectivity = azurerm

    azurerm.management   = azurerm

  }

root_parent_id = data.azurerm_client_config.core.tenant_id

  root_id        = ""

  root_name      = ""

}</code></pre>

Make sure to give root_id and root_name values that are suitable to your organization. For example, if you work for the "FlyMeToTheCloud" company, you can set it to something like:

  root_id        = "Fly"
  root_name      = "FlyMeToTheCloud"

On VSCode open a terminal and make sure that you cd to the directory that contains your main.tf file. Then, type "az login" to connect to your Azure tenant.



Now, run "terraform init" to initialize Terraform for the new configuration.



Next, run "terraform plan" to preview the changes that the new configuration would apply to Azure. Take a look to the output of this command. In this case, 183 new resources will be created in Azure. This is expected. You can scroll up in the terminal window to review the detailed list of what will be deployed.



Finally, run "terraform apply" to deploy the changes to Azure. When asked, type yes to approve the planned changes. Once completed, review the number of resources that have been deployed to Azure.



Navigate to the Azure Portal and review the applied changes. For example, open the Management Groups page and review the new management groups. Note that there is a new MG called "FlyMeToTheCloud" with the ID "Fly". This is the "Intermediate Root" MG.



You can also review the policy assignments created by our deployment. On the Azure Portal, navigate to "Policy", click "Assignments" and set the "Scope" to the Intermediate Root MG which in our case is "FlyMeToTheCloud". You can have a look at all the actual policies assigned at the dedicated page on GitHub.



The above changes should not produce any charges to your Azure subscriptions. After all, we just created some management groups and applied a few policies, which are all provided for free. Furthermore, these should not impact any resources (e.g. Virtual Machines) that you might have already created, since we did not move any subscriptions under these MGs. We will discuss this point in a future post.

How many times have you tried to write a script that needs to open a connection to a remote machine such as a SFTP server? Most probably this machine will ask you to authenticate before uploading any files. In order to perform authentication, you would need to provide the username and password for the connection to be established. Normally, you would have to store these credentials somewhere on the disk. If you were smart enough to avoid storing such sensitive information in plain text, you would use the "ConvertTo-SecureString" cmdlet in order to encrypt the password. In most cases, only the user who encrypted the password could decrypt it.

If you are looking for a way to avoid storing these credentials on your machine, let me introduce you to Azure Key Vault. It's a service that can help you store your keys, secrets and certificates in a secure manner, either using Microsoft-managed keys or customer-managed keys backed by a Hardware Security Module (HSM). That said, you could safely store your script passwords into Azure and retrieve them temporarily during runtime, as long as the script execution lasts.

As you may expect, Azure Key Vault implements Role Based Access Control policies in order to granularly define who can access it and which secrets they can retrieve. You might wonder, why should I use a Key Vault if I need to authenticate against it, using stored credentials? Well, the solution to that is provided by his majesty, Azure Active Directory and its System Assigned Managed Identity feature. This way, your Azure VM can connect to Azure Key Vault without having to store any credentials on the disk or the script code. Let me show you how that works.

First, go to the Azure portal, look for your Virtual Machine where your script runs and go the "Identity blade". In the "System Assigned" tab set "Status" to "On" and click "Save". What happens in the background is that your Azure VM receives a service principal in Azure Active Directory and you can use it in order to allow your VM to access any Azure resource that supports Azure AD authentication.

Next, let's create our Azure Key vault. In Azure Portal search for "Key Vault" and then choose "Create Key Vault". In the "Create Key Vault" page we just need a few basics such as Resource Group name, Key Vault name, Region and Pricing Tier. Here we choose "Standard". We can opt-for "Premium" in case we prefer to use a Hardware Security Module (HSM) to protect our keys. Not our case, so we go with "Standard".
We will not examine the rest of the tabs in this article, let's go straight away to "Review + create" and then "Create".

Now we need to perform some configuration on our newly created Key Vault. First, we need to add the password that we are going to retrieve later. In the Azure Portal, look for your Key Vault, move to the "Secrets" blade, and choose "Generate\Import". Give a name to your secret such as "ftpadmin" (this could be your username) and type the actual password in the "Value" field. Make sure "Enabled" is set to "Yes" and click "Create".

Back to our Key Vault page, go to the "Access policies" blade and choose "Add access policy". Since we are going to retrieve a secret, we need to authorize our Azure VM for this action. In the "Secret permissions" menu choose at least "Get". In the "Select principal" option click on "None selected" and now type the name of your VM to find the managed identity we created earlier. Choose your VM, click "Select". Finally, click "Add".
Caution: Once you return to the "Access policies" blade, do not forget to click "Save"

So now that we have prepared everything, let's take a look at the PowerShell code. What you need to do is to add a few lines to your script that will help you retrieve the secret from the Key Vault during execution. Of course, you will need to run this on the Azure VM for which you created the system-assigned managed identity a few lines above.

#Connect to the Azure Instance Metadata Service (IMDS)
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fvault.azure.net' -Method GET -Headers @{Metadata="true"}

#Convert the JSON formatted response to a PowerShell object 

$content = $response.Content | ConvertFrom-Json

#Store the token into a variable

$KeyVaultToken = $content.access_token

#Connect to the Azure Key Vault passing the token in the HTTP headers 

((Invoke-WebRequest -Uri https://<your_key_vault_URL>/secrets/<your_secret_name>?api-version=2016-10-01 -Method GET -Headers @{Authorization="Bearer $KeyVaultToken"}).content | ConvertFrom-Json).value</pre>

What happens in the code above is that the Azure VM connects to the Azure Instance Metadata Service (IMDS) in order to retrieve an access token. The response from IMDS is in JSON format so we need to use "ConvertFrom-Json" in order to extract the token string. Then, we make another request to our Key Vault incorporating the access token into the HTTP headers. This way, Key Vault will evaluate this token against the access policies and decide whether to allow access or not. 
Note that in the last command you will need to fill in you Key Vault URL (available in the Key Vault's overview blade) as well as the secret name (ftpadmin in our example). 



You can see the execution output in the following image. 

In my first post I decided to talk about two technologies which, when combined, can give you a lot of power managing your VMs in the cloud.

First one is PowerShell. Generally, I prefer to use it for my daily tasks. I have also used it a lot in my career, in order to help organizations get rid of long, boring, GUI-driven processes and replace them with code that works every time, with the same result but, most importantly, way faster.
Second one is Microsoft Azure. I started working with Azure about two years ago and I can say that it has opened new fields for me enabling rapid and consistent deployments, helping to stay up-to-date and take advantage of the latest and coolest technologies in the IT industry.

So, this one is dedicated to Azure Automation State Configuration. It is a Configuration Management tool that can help you push declarative configurations to your servers, both in the cloud and on-premises, either Windows or Linux. It is based on PowerShell Desired State Configuration (DSC) which became available with PowerShell 4.0 in Windows Server 2012 R2. PowerShell DSC works in two modes, "push" and "pull". In the "push" mode the administrator runs a command that "pushes" a configuration against one or more servers. In the "pull" mode, the administrator uploads all configurations in a central repository called the "pull server". Then, each server contacts the "pull server" on specified intervals to see if there is any new configuration for him to "pull" and apply.

Azure Automation State Configuration offers you a "pull-server-as-a-service", which means that you can upload your configurations to Azure, register your servers to the service and monitor their compliance status through a reporting tool. In order to set things up, you will need the following components:

• an Azure Automation Account
• some DSC code
• your own servers

Create an Automation Account

Creating an Automation Account is as easy as searching for "automation" in the Azure portal. Choose "Automation Accounts" then click "Create automation account". You will need to specify a name for your automation account, a resource group and a location.

You can also create your automation account using the following Azure PowerShell cmdlet:

New-AzAutomationAccount -Name Automation1 -ResourceGroupName rg1 -Location westeurope

Import a DSC configuration

Now that you have created your Automation Account, it's time to populate with some configurations, that is, the actual PowerShell code. Look for your newly created Automation Account "Automation1" and navigate to "State configuration (DSC) / Configurations" and then click "Add".

Choose to upload a DSC configuration file from your hard drive and click "OK".

There are already tons of configuration files available on the Internet. You can grab a very basic one from my Github page.

You can also import a configuration file using Azure PowerShell.

Import-AzAutomationDscConfiguration -AutomationAccountName "Automation1" -ResourceGroupName "rg1" -SourcePath ".\IISConfig.ps1" -Published

Next step is to compile your configuration. Back in the Azure Portal, on the "Configurations" tab, choose your configuration and in the next window click "Compile".

To achieve this with Azure PowerShell just run the following cmdlet:

Start-AzAutomationDscCompilationJob -ConfigurationName IISConfig -ResourceGroupName rg1 -AutomationAccountName Automation1

Now that our code is ready, it is time to assign the configuration to our server. For this purpose I have already created an Azure VM called "vm1".
In the "State Configuration (DSC) blade, navigate to "Nodes" and click "Add".

Choose your VM from the list and click "Connect".

In the last window, you can make a few choices such as:

• Node configuration name: We will choose "IISConfig.IsWebServer" which refers to the part of our PowerShell code that installs IIS on our Windows Server.
• Refresh Frequency: Refers to how often the VM checks for new configurations. Default is 30 minutes.
• Configuration Mode: You can choose how to apply the configuration to your server. We choose "ApplyAndMonitor".
• Configuration Mode Frequency: It specifies how often the "Configuration Mode" is applied. Works only with " ApplyAndAutoCorrect" and "ApplyAndMonitor". Default is 15 minutes.

Apart from using Azure Portal, you can also register your VMs as DSC nodes using the following Azure PowerShell cmdlet:

Register-AzAutomationDscNode -AutomationAccountName Automation1 -AzureVMName vm1 -ResourceGroupName rg1 -NodeConfigurationName "IISConfig.IsWebServer" -ConfigurationMode ApplyAndMonitor -RefreshFrequencyMins 30 -ConfigurationModeFrequencyMins 15

Once you click OK, the "Microsoft.Powershell.DSC" extension will be installed to your VM. As soon as the registration process is completed, the configuration will be applied to your server. In our case, IIS will be installed. You can confirm that by checking IIS installation status on your VM.

You can also use Azure Portal to ensure that your nodes are compliant with the assigned configurations.

As a last step, I will show you what happens if IIS gets uninstalled from the server. First, let's remove the role using Windows PowerShell

Now let's go to Azure Portal, "State Configuration (DSC)" blade.

Your VM now appears as "Not compliant" with the assigned configuration. Remember that during node registration, we chose "ApplyAndMonitor" as the configuration mode. This means that configuration was applied only once, during the initial registration.

Had we opted for "ApplyAndAutoCorrect", the DSC mechanism would have reinstalled IIS, ensuring that our server always stays compliant with the configuration.