Article explains how to send Spring Boot and Netflix Servo metrics to AWS CloudWatch. Morover it describes mechanisms making it happen. It also mentions problems I run into trying to do the same with Spring Boot and Spectator.

The article desribes situation before and is not aplicable to Spring Boot 2.0 GA.
For metrics in Spring Boot 2.0 check here

TL;DR

If you are not interested in how and why and just want to make Spring Boot work with AWS CloudWatch do the following:

  • add dependencies (here build.gradle)
dependencies {
    compile('org.springframework.cloud:spring-cloud-starter-aws')
    compile('org.springframework.cloud:spring-cloud-aws-actuator')
    compile('com.netflix.servo:servo-core')
    compile('org.aspectj:aspectjweaver')
}
  • set namespace name using property (in application.properties)
cloud.aws.cloudwatch.namespace=m3trics

That’s it unless you are interested in why and how it works - then please continue reading :). You can also check and follow everything in the working code

 

SpringBoot Actuator

It’s super easy to enable some metrics with Spring Boot Actuator

So let’s create SpringBoot project with two dependencies:

compile('org.springframework.boot:spring-boot-starter-web')
compile('org.springframework.boot:spring-boot-starter-actuator')

and disable security for actuator endpoints in application.properties

management.security.enabled=false

Now launch the app and hit http://localhost:8080/application/metrics

and enjoy metrics like: mem, processors, heap, etc…

To find providers for presented metrics check org.springframework.boot.actuate.endpoint.PublicMetrics interface and it’s class hierarchy.

 

CounterService & GaugeService

Of the PublicMetrics providers MetricReaderPublicMetrics is special as it reads all metrics registered by CounterService and GaugeService, which in short

CounterService - registers tracks just inreasing a counter (so can be used to number of requests, pages visits, etc)

GaugeService - can store any value (it’s not incremental - just set), so it’s application may vary from from measuring time to showing info about threads, conntections, etc.

These two are used in many places, like gathering metrics with every request to any endpoint. Even for http://localhost:8080/application/metrics we have:

gauge.response.application.metrics: 0.0
counter.status.200.application.metrics: 7

where application.metrics is taken from url path -> /application/metrics

counter.status.200 means number of hits to this particular endpoint with response code = 200

gauge.response - is last response time

 

The inner workings are simple. With spring-boot-starter-actuator by MetricFilterAutoConfiguration we added MetricsFilter which basically is javax.servlet.Filter submiting metrics by CounterService and CounterService for every request.

CounterService and GaugeService may also be used to create metrics on our own, like in:

@RestController
@RequestMapping("/favorites")
class FavoritesNumberController {

    private final GaugeService gaugeService;

    FavoritesNumberController(GaugeService gaugeService) {
        this.gaugeService = gaugeService;
    }

    @GetMapping
    void favoriteNumber(@RequestParam Double number) {
      gaugeService.submit("favoriteNumber", number);
    }
}

Now hit http://localhost:8080/favorites?number=11 and get not only:

gauge.response.favorites: 24.0,
counter.status.200.favorites: 3

submitted by MetricsFilter but also our custom metric:

gauge.favoriteNumber: 11.0

CounterService and GaugeService are interfaces and by default they use in memory objects (CounterBuffers, GaugeBuffers) to store metrics.

To send it to the outside world MetricExporters are used which in separate thread (MetricExporters.ExportRunner) will send the counter and gauge metrics wherever we want. MetricExportAutoConfiguration tells us that to make it work at least one MetricWriter need to be present.

None is registered but deafult but even with spring-boot-starter-actuator we got components to write metrics as MBeans, send them to StatsD (front-end proxy for the Graphite/Carbon metrics server) or store them in Redis (check different implementations of org.springframework.boot.actuate.metrics.writer.MetricWriter).

Nothing for CloudWatch here, but we can easilly get there.

 

Sending Metrics to AWS CloudWatch

Now to be able to write our metrics to CloudWatch we need CloudWatchMetricWriter and we get it by adding

compile('org.springframework.cloud:spring-cloud-aws-actuator')
compile('org.springframework.cloud:spring-cloud-starter-aws')

to our dependencies. In the second dependency we’ll find CloudWatchMetricAutoConfiguration which registers our CloudWatchMetricWriter.

If we check CloudWatchMetricAutoConfiguration

@Configuration
@Import(ContextCredentialsAutoConfiguration.class)
@EnableConfigurationProperties(CloudWatchMetricProperties.class)
@ConditionalOnProperty(prefix = "cloud.aws.cloudwatch", name = "namespace")
@ConditionalOnClass(name = {"com.amazonaws.services.cloudwatch.AmazonCloudWatchAsync",
        "org.springframework.cloud.aws.actuate.metrics.CloudWatchMetricWriter"})
public class CloudWatchMetricAutoConfiguration {
(...)

we will see that one more piece is missing: cloud.aws.cloudwatch.namespace so let’s set it in application.properties

cloud.aws.cloudwatch.namespace=m3trics

Now application can be deployed to the AWS Cloud, or run locally, if we have proper AWS credentials configured. (For different types of providing AWS credentials look at DefaultAWSCredentialsProviderChain class). More over, if you are running app locally also add the following property:

cloud.aws.region.static=us-east-1

(or other region that you use) as Spring Cloud won’t be able to figure it out if not running in EC2 instance.

Now access different endpoints in app and with few seconds delay you will be able to observe our counter / gauge metrics in AWS CloudWatch.

Log to the AWS Console, and go to CloudWatch -> m3trics (which we set as namespace)

Registered Metrics namespace Registered Metrics namespace in CloudWatch

 

Spring Boot basic metrics in CloudWatch Spring Boot basic metrics in CloudWatch

 

Netflix Servo Metrics

All of this is of course only half of the story. With Spring Cloud we will use a lot of Netflix libraries, and these do not use Spring Cloud’s CounterService / GaugeService but generate metrics on their own using Netflix Servo and Spectator collection libraries. (you will find some info about these here, main point being - Spectator is newer and should replace old Servo :))

Let’s try with Hystrix and add dependency to our project:

compile('org.springframework.cloud:spring-cloud-starter-hystrix')

With Hystrix we also got spring-cloud-netfix-core library and inside we find MetricsInterceptorConfiguration, which is loaded conditionally on servo package present, so let’s add it

compile('com.netflix.servo:servo-core')

We want to gather metrics from RestTemplate as well and MetricsInterceptorConfiguration specifies that this one is dependent on aspectjweaver, so let’s add it as well.

compile('org.aspectj:aspectjweaver')

With ServoMetricsAutoConfiguration we got ServoMetricServices which is both CounterService and GaugeService implementation. So from now on, all gathered servo metrics will be sent to CloudWatch!

Let’s check some Hystrix in action. Just enable it by @EnableCircuitBreaker, and create some hystrix traffic, like:

@Service
class SomeTrafficGenerator {

    private final RestTemplate restTemplate;

    SomeTrafficGenerator(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @Scheduled(fixedDelay = 5000)
    @HystrixCommand
    void makeSomeHit() {
        restTemplate.getForObject("https://www.google.pl", String.class);
    }

}

Run the application one more time and enjoy plenty of new metrics in CloudWatch!

Hystrix metrics in CloudWatch

 

Spring Boot and Spectator metrics

With so many successes so far you might be tempted to do the same with Spectator by adding

compile('org.springframework.cloud:spring-cloud-starter-netflix-spectator')

dependency, but I don’t think Spring Boot is ready to work with it.

Add spectator as dependency and ServoMetricsAutoConfiguration will no longer be loaded (@ConditionalOnMissingClass(“com.netflix.spectator.api.Registry”)), so no ServoMetricReader will be registered, so MetricsExporter will not work. (there is SpectatorMetricReader in spring-cloud-netflix-spectator library but for some reason it’s not registered by SpectatorMetricsAutoConfiguration).

 

In the time of wriritng the newest Spring Cloud library is Finchley.M2 (and Finchley is the only version working with Spring Boot 2.0) - so maybe everything will work with GA version.

 

Again you may check the code here