AEM Performance Optimization CPU/HEAP Status

Problem Statement:

As a developer or user, I would like to make an informed decision by fetching the AEM system CPU/Heap status before or while running a process.

Introduction:

Use cases for developers:

  1. Infinite loops – a coding error
  2. Garbage collection is not handled – unclosed streams
  3. An exception like out of bound issue
  4. Heap size issue – saving loads of data or declaring/manipulating too many strings

Java MX Bean is an API that provides detailed information on JVM CPU/MEM (Heap) status.

A platform MXBean is a managed bean that conforms to the JMX Instrumentation Specification and only uses a set of basic data types. A JMX management application and the platform MBeanServer can interoperate without requiring classes for MXBean specific data types. The data types being transmitted between the JMX connector server and the connector client are open types and this allows interoperation across versions. See the specification of MXBeans for details.

Each platform MXBean is a PlatformManagedObject and it has a unique ObjectName for registration in the platform MBeanServer as returned by the getObjectName method.

As developer before running or while running any bulk process or schedulers, it’s always better to get system information.

ThrottledTaskRunnerStats Service:

Create ThrottledTaskRunnerStats service as shown below:

package com.aem.operations.core.services;

import javax.management.InstanceNotFoundException;
import javax.management.ReflectionException;

/**
 * Private interface for exposing ThrottledTaskRunner stats
 * **/
public interface ThrottledTaskRunnerStats {
    /**
     * @return the % of CPU being utilized.
     * @throws InstanceNotFoundException
     * @throws ReflectionException
     */
    double getCpuLevel() throws InstanceNotFoundException, ReflectionException;

    /**
     * The % of memory being utilized.
     * @return
     */
    double getMemoryUsage();

    /***
     * @return the OSGi configured max allowed CPU utilization.
     */
    double getMaxCpu();

    /***
     * @return the OSGi configured max allowed Memory (heap) utilization.
     */
    double getMaxHeap();

    /**
     * @return the max number of threads ThrottledTaskRunner will use to execute the work.
     */
    int getMaxThreads();
}

ThrottledTaskRunnerImpl Service Implementation:

Create ThrottledTaskRunnerImpl service implementation as shown below:

package com.aem.operations.core.services.impl;

import java.lang.management.ManagementFactory;
import javax.management.Attribute;
import javax.management.AttributeList;
import javax.management.AttributeNotFoundException;
import javax.management.InstanceNotFoundException;
import javax.management.MBeanException;
import javax.management.MBeanServer;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.management.ReflectionException;
import javax.management.openmbean.CompositeData;
import com.aem.operations.core.services.ThrottledTaskRunnerStats;
import org.osgi.service.component.annotations.Activate;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Modified;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@Component(service = ThrottledTaskRunnerStats.class, immediate = true, name = "Throttled Task Runner Service Stats")
public class ThrottledTaskRunnerImpl implements ThrottledTaskRunnerStats {

    private static final Logger LOGGER = LoggerFactory.getLogger(ThrottledTaskRunnerImpl.class);
    private final MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
    private ObjectName osBeanName;
    private ObjectName memBeanName;

    @Activate
    @Modified
    protected void activate() {
        try {
            memBeanName = ObjectName.getInstance("java.lang:type=Memory");
            osBeanName = ObjectName.getInstance("java.lang:type=OperatingSystem");
        } catch (MalformedObjectNameException | NullPointerException ex) {
            LOGGER.error("Error getting OS MBean (shouldn't ever happen) {}", ex.getMessage());
        }
    }

    @Override
    public double getCpuLevel() throws InstanceNotFoundException, ReflectionException {
        // This method will block until CPU usage is low enough
        AttributeList list = mbs.getAttributes(osBeanName, new String[]{"ProcessCpuLoad"});

        if (list.isEmpty()) {
            LOGGER.error("No CPU stats found for ProcessCpuLoad");
            return -1;
        }

        Attribute att = (Attribute) list.get(0);
        return (Double) att.getValue();
    }

    @Override
    public double getMemoryUsage() {
        try {
            Object memoryusage = mbs.getAttribute(memBeanName, "HeapMemoryUsage");
            CompositeData cd = (CompositeData) memoryusage;
            long max = (Long) cd.get("max");
            long used = (Long) cd.get("used");
            return (double) used / (double) max;
        } catch (AttributeNotFoundException | InstanceNotFoundException | MBeanException | ReflectionException e) {
            LOGGER.error("No Memory stats found for HeapMemoryUsage", e);
            return -1;
        }
    }

    @Override
    public double getMaxCpu() {
        return 0.75;
    }

    @Override
    public double getMaxHeap() {
        return 0.85;
    }

    @Override
    public int getMaxThreads() {
        return Math.max(1, Runtime.getRuntime().availableProcessors()/2);
    }
}

CPU Status Servlet:

Create a CpuStatusServlet based on the path as shown below: 

package com.aem.operations.core.servlets;

import java.io.IOException;
import java.io.PrintWriter;
import java.text.MessageFormat;
import javax.management.InstanceNotFoundException;
import javax.management.ReflectionException;
import javax.servlet.Servlet;
import javax.servlet.ServletException;
import com.aem.operations.core.services.ThrottledTaskRunnerStats;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.gson.Gson;
import com.google.gson.JsonObject;

@Component(service = { Servlet.class }, property = { "sling.servlet.paths=" + CpuStatusServlet.RESOURCE_PATH,
        "sling.servlet.methods=POST" })
public class CpuStatusServlet extends SlingAllMethodsServlet{

    private static final Logger LOGGER = LoggerFactory.getLogger(CpuStatusServlet.class);

    private static final long serialVersionUID = 1L;
    public static final String RESOURCE_PATH = "/bin/cpustatust";
    private static final String MESSAGE_FORMAT = "{0,number,#%}";

    @Reference
    private transient ThrottledTaskRunnerStats ttrs;

    @Override
    protected void doGet(SlingHttpServletRequest request, SlingHttpServletResponse response)
            throws ServletException, IOException {
        JsonObject jsonResponse = getSystemStats();
        try (PrintWriter out = response.getWriter()) {
            out.print(new Gson().toJson(jsonResponse));
        }
    }

    private JsonObject getSystemStats() {
        JsonObject json = new JsonObject();
        try {
            json.addProperty("cpu", MessageFormat.format(MESSAGE_FORMAT, ttrs.getCpuLevel()));
        } catch (InstanceNotFoundException | ReflectionException e) {
            LOGGER.error("Could not collect CPU stats {}", e.getMessage());
            json.addProperty("cpu", -1);
        }
        json.addProperty("mem", MessageFormat.format(MESSAGE_FORMAT, ttrs.getMemoryUsage()));
        json.addProperty("maxCpu", MessageFormat.format(MESSAGE_FORMAT, ttrs.getMaxCpu()));
        json.addProperty("maxMem", MessageFormat.format(MESSAGE_FORMAT, ttrs.getMaxHeap()));
        return json;
    }
}

Frontend:

Make a request to a servlet from frontend using AJAX call as shown below:

function getStatus(showStatus) {
   $.ajax({
      // add the servlet path
      url: "/bin/cpustatust",
      method: "GET",
      async: true,
      cache: false,
      contentType: false,
      processData: false
   }).done(function (data) {
      if (data) {
         data = JSON.parse(data);
         $("#table-body").append(`<tr>
						<td>${data.cpu}/${data.maxCpu}</td>
			            <td>${data.mem}/${data.maxMem}</td>
		           		 </tr>`);
      } else {
         $(".modal").hide();
         showStatus = false;
         ui.notify("Error", "Unable to get the status", "error");
      }
   }).fail(function (data) {
      $(".modal").hide();
      showStatus = false;
      if (data && data.responseJSON && data.responseJSON.message) {
         ui.notify("Error", data.responseJSON.message, "error");
      } else {
         //add error message
         ui.notify("Error", "Unable to get the status", "error");
      }
   });
   if (showStatus) {
      setTimeout(() => {
         emptyResults();
         getStatus(true);
      }, 2000);
   }
}
function emptyResults() {
   $("#table-body").empty();
}
<table id="table-data">
   <thead id="table-head">
   <tr>
      <th>CPU Usage</th>
      <th>Memory (Heap) Usage</th>
   </tr>
   </thead>
   <tbody id="table-body">
   </tbody>
</table>

The output of the status:

CPU Memory status result

For more information on Throttled Task runner-based performance optimization please visit:

  1. AEM Performance Optimization Scheduler
  2. AEM Performance Optimization Workflow
  3. AEM Performance Optimization Replication

AEM Performance Optimization Activation/Replication

Problem Statement:

AEM Bulk Replication allows you to activate a series of pages and/or assets.

How can we make sure workflow won’t impact AEM performance (CPU or Heap memory) / throttle the system?

Introduction:

AEM bulk replication or activation is performed on a series of pages and/or assets. We usually perform bulk replication on tree structures or lists of paths.

Use case:

  1. MSM (Multi-site management) – rolling out a series of pages or site
  2. Editable template – add/remove new components on template structure and activate existing pages
  3. Bulk Asset ingests into the system
  4. Bulk redirect//vanity path update

ACS Commons Throttled Task Runner is built on Java management API for managing and monitoring the Java VM and can be used to pause tasks and terminate the tasks based on stats.

Throttled Task Runner (a managed thread pool) provides a convenient way to run many AEM-specific activities in bulk it basically checks for the Throttled Task Runner bean and gets current running stats of the actual work being done.

OSGi Configuration:

The Throttled Task Runner is OSGi configurable, but please note that changing configuration while work is being processed results in resetting the worker pool and can lose active work.

Throttled Task Runner OSGi

Max threads: Recommended not to exceed the number of CPU cores. Default 4.

Max CPU %: Used to throttle activity when CPU exceeds this amount. Range is 0..1; -1 means disable this check.

Max Heap %: Used to throttle activity when heap usage exceeds this amount. Range is 0..1; -1 means disable this check.

Cooldown time: Time to wait for CPU/MEM cooldown between throttle checks (in milliseconds)

Watchdog time: Maximum time allowed (in ms) per action before it is interrupted forcefully.

JMX MBeans

Throttled Task Runner MBean

This is the core worker pool. All action managers share the same task runner pool, at least in the current implementation. The task runner can be paused or halted entirely, throwing out any unfinished work.

Throttled task runner JMX
Throttled task runner JMX

How to use ACS Commons throttled task runner

Add the following dependency to your pom

Create a custom service or servlet as shown below:

Throttled Replication

inside the custom workflow process method check for the Low CPU and Low memory before starting your task to avoid performance impact on the system.

For bulk replication (publish/unpublish/delete) assets or pages, please refer to the AEM Operation Replication Tool

For best practices on the AEM servlet please refer to the link.

AEM Performance Optimization Workflow

Problem Statement:

AEM Workflows allow you to automate a series of steps that are performed on (one or more) pages and/or assets.

How can we make sure workflow won’t impact AEM performance (CPU or Heap memory) / throttle the system?

Introduction:

AEM Workflows allow you to automate a series of steps that are performed on (one or more) pages and/or assets.

For example, when publishing, an editor has to review the content – before a site administrator activates the page. A workflow that automates this example notifies each participant when it is time to perform their required work:

  1. The author applies the workflow to the page.
  2. The editor receives a work item that indicates that they are required to review the page content. When finished, they indicate that their work item is complete.
  3. The site administrator then receives a work item that requests the activation of the page. When finished, they indicate that their work item is complete.

Typically:

  • Content authors apply workflows to pages as well as participate in workflows.
  • The workflows that you use are specific to the business processes of your organization.

ACS Commons Throttled Task Runner is built on Java management API for managing and monitoring the Java VM and can be used to pause tasks and terminate the tasks based on stats.

Throttled Task Runner (a managed thread pool) provides a convenient way to run many AEM-specific activities in bulk it basically checks for the Throttled Task Runner bean and gets current running stats of the actual work being done.

OSGi Configuration:

The Throttled Task Runner is OSGi configurable, but please note that changing configuration while work is being processed results in resetting the worker pool and can lose active work.

Throttled task runner OSGi

Max threads: Recommended not to exceed the number of CPU cores. Default 4.

Max CPU %: Used to throttle activity when CPU exceeds this amount. Range is 0..1; -1 means disable this check.

Max Heap %: Used to throttle activity when heap usage exceeds this amount. Range is 0..1; -1 means disable this check.

Cooldown time: Time to wait for CPU/MEM cooldown between throttle checks (in milliseconds)

Watchdog time: Maximum time allowed (in ms) per action before it is interrupted forcefully.

JMX MBeans

Throttled Task Runner MBean

This is the core worker pool. All action managers share the same task runner pool, at least in the current implementation. The task runner can be paused or halted entirely, throwing out any unfinished work.

Throttled task runner JMX
Throttled task runner JMX

How to use ACS Commons throttled task runner

Add the following dependency to your pom

<dependency>
    <groupId>com.adobe.acs</groupId>
    <artifactId>acs-aem-commons-bundle</artifactId>
    <version>5.0.4</version>
    <scope>provided</scope>
</dependency>

Create a custom workflow and call the service as shown below:

Throttled Workflow

inside the custom workflow process method check for the Low CPU and Low memory before starting your task to avoid performance impact on the system.

For all the bulk workflow executions on pages/assets, it’s recommended to use ACS commons bulk workflow.

For best practices on workflow please refer to the link

AEM Performance Optimization Scheduler

Problem Statement:

AEM Schedulers are commonly used to run bulk tasks at an off time (nonbusiness hours) and some tasks are run periodically to fetch results cached and fetched by the front end.

How can we make sure bulk tasks won’t impact AEM performance (CPU or Heap memory) / throttle the system?

Introduction:

In Computer Software, Scheduling is a paradigm of planning for the execution of a task at a certain point in time and it can be broadly classified into two types:

1. Scheduled Task – executing once at a particular future point in time

2. Frequent scheduling – repeat periodically at a fixed interval

The use case for the scheduler:

1. Sitemap generation

2. Synching product data from AEM Commerce

3. Fetch DB content and place it in the repository to be picked up by frontend and in turn cached in Dispatcher

4. Fetch Stats or reports and place them in the repository to be picked up by frontend and in turn cached in Dispatcher

ACS Commons Throttled Task Runner is built on Java management API for managing and monitoring the Java VM and can be used to pause tasks and terminate the tasks based on stats.

Throttled Task Runner (a managed thread pool) provides a convenient way to run many AEM-specific activities in bulk it basically checks for the Throttled Task Runner bean and gets current running stats of the actual work being done.

OSGi Configuration:

The Throttled Task Runner is OSGi configurable, but please note that changing configuration while work is being processed results in resetting the worker pool and can lose active work.

Throttled task runner OSGi

Max threads: Recommended not to exceed the number of CPU cores. Default 4.

Max CPU %: Used to throttle activity when CPU exceeds this amount. Range is 0..1; -1 means disable this check.

Max Heap %: Used to throttle activity when heap usage exceeds this amount. Range is 0..1; -1 means disable this check.

Cooldown time: Time to wait for CPU/MEM cooldown between throttle checks (in milliseconds)

Watchdog time: Maximum time allowed (in ms) per action before it is interrupted forcefully.

JMX MBeans

Throttled Task Runner MBean

This is the core worker pool. All action managers share the same task runner pool, at least in the current implementation. The task runner can be paused or halted entirely, throwing out any unfinished work.

Throttled task runner JMX

How to use ACS Commons throttled task runner

Add the following dependency to your pom

<dependency>
     <groupId>com.adobe.acs</groupId>
     <artifactId>acs-aem-commons-bundle</artifactId>
     <version>5.0.4</version>
     <scope>provided</scope>
 </dependency>

Create a scheduler and call the service as shown below:

Throttled Scheduler call

inside the run, method check for the Low CPU and Low memory before starting your task to avoid performance impact on the system

For the best practices on AEM Scheduler please refer to the link