Packaging Amazon CloudWatch Agent for NixOS

DevOps

by Pim Snel/ on 1 Mar 2024

Packaging Amazon CloudWatch Agent for NixOS

For an internal research project, we have installed NixOS on an AWS EC2 instance. NixOS is ideal in an Infrastructure as Code setup combined with Terraform and AWS. In a series of upcoming blogs, I will delve deeper into the benefits of NixOS in the AWS cloud. This blog focuses on NixOS and the Amazon CloudWatch Agent, or rather the absence of this software for NixOS.

Spoiler… We have built a Nix Flake with which everyone can now install Amazon CloudWatch Agent on NixOS.

The research project involved creating a VPC with a number of EC2 instances that are configured by NixOS. During this project, we encountered an unexpected problem. An application on an EC2 instance caused memory issues. So far, nothing new, but when we wanted to analyze the memory usage in the EC2 instance with CloudWatch Metrics, we discovered that Amazon’s CloudWatch Agent software is not available for NixOS.

AWS EC2 instances still are well supported by NixOS. The SSM agent is available as an Nix official package, as well as the special ENA network drivers.

Internally and Externally Available Metrics

A hotelroom is a good metaphore for what CloudWatch can see and cannot see. The hotel management can see how much water a hotel guest uses in their room. However, they cannot see if the guest is showering or filling up the bath.

Photo by Roberto Nickson on Unsplash

Without running an agent on the EC2 instance, CloudWatch can display a number of standard metrics. CPU usage, Disk IO, Network IO, are metrics that AWS can display for each machine. However, some essential metrics can only be viewed from within the instance. AWS cannot see what filesystem is mounted inside an EC2 instance, or how much swap space is still available. And also, memory is a blind spot for AWS. Memory is managed by the Linux kernel and is therefore outside the scope of AWS as a host. The internally available metrics must be published to CloudWatch through an Agent. Amazon CloudWatch Agent does this.

standard available metrics

Luckily, the CloudWatch Agent is an Open Source project written in Go so it is not difficult to write a NixOS Package. We have not created a PR yet so for now the CloudWatch Agent is not available through the official NixOS channels. Those who want to use CloudWatch on NixOS can use our CloudWatch Agent Flake. We will briefly explain how to implement this flake. The only requirement is that your Nix setup is based on Nix Flakes.

Install Amazon CloudWatch Agent on NixOS

In the inputs of your flake, add the Amazon CloudWatch Agent flake.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
    amazon-cloudwatch-agent.url = "github:mipmip/amazon-cloudwatch-agent-nix";
  }
}

Then you will need to import the module and also add the cloudwatch agent software to the system packages.

Below is an example setup.

{
  description = "NixOS configuration Amazon CloudWatch Agent";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.11";
    amazon-cloudwatch-agent.url = "github:mipmip/amazon-cloudwatch-agent-nix";
  };

  outputs = { self, nixpkgs, amazon-cloudwatch-agent }:
    let
      system = "x86_64-linux";

      amazon-cloudwatch-module = amazon-cloudwatch-agent.nixosModules.default;
      amazon-cloudwatch-config = {
        services.amazon-cloudwatch-agent.enable = true;
        environment.systemPackages = [
         amazon-cloudwatch-agent.packages."${system}".amazon-cloudwatch-agent
        ];
      };

    in {
      nixosConfigurations."<hostname>" = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          amazon-cloudwatch-module
          amazon-cloudwatch-config
          ./configuration.nix
        ];
      };
    };
}

Give CloudWatch Agent permission to publish to CloudWatch

Once the agent is installed, you just need to make sure it has permission to publish its metrics to CloudWatch. You grant this permission by adding a policy to the IAM Instance Profile.

Below is an example piece of Terraform code on how to add this to your EC2 profile.

resource "aws_iam_instance_profile" "ssm-access-iam-profile" {
  name = "ec2_profile"
  role = aws_iam_role.ssm-access-iam-role.name
}

resource "aws_iam_role" "ssm-access-iam-role" {
  name        = "ssm-access-role"
  description = "The role to access EC2 with SSM"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect    = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "cloudwatch-policy" {
  role       = aws_iam_role.ssm-access-iam-role.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

About NixOS

NixOS is a renowned and mature Linux Distribution based on a revolutionary concept to describe OS configuration using a functional programming language called Nix. NixOS cannot be easily described in a few lines, but from the perspective of a DevOps engineer, NixOS can be seen as a Linux OS in which the functionality of Ansible is built-in by default. You do not install NixOS. You create a configuration and then deploy it.

About Amazon CloudWatch

CloudWatch is AWS’s standard monitoring service. Through CloudWatch, you can centrally analyze log files and metrics. You can also set up CloudWatch to generate alarms when certain metrics reach their threshold values. CloudWatch is a core component of Technative’s ACOS product.

About TechNative ACOS

Part of TechNative’s Standard Base is TechNative’s Advanced Cloud Observability System. The system leverages on AWS CloudWatch, to give complete infrastructure visibility. In the integrated Messaging Hub, connections can be made with various alarm management systems or messaging apps, such as OpsGenie, PagerDuty or Slack. Read more about ACOS in the TechNative DevOps Whitepaper.