AEM Link Checker Introduction

AEM link checker is used to validate all internal and external links available on the page. 

The main purpose of developing link checker is that content author should not worry about bad or broken links on publish environment , it also allow authors to view list of all valid and invalid links available on his website at a single place


AEM External link checker:-
                    AEM Link Checker is based on an event Handler and gets triggered on creates and updates for /content and its child nodes. 

All content under selected root path is parsed and links are validated. All the validation of links is done asynchronously in the background and the HTML is updated based on verification results. 

Note:- If you are having huge repository (/content), that includes frequent updation of links. Then it is not advised to use link checker due to performance issues. 
As it gets triggered periodically and traverse the whole repository for validating links. This may cause slowness in your author instance.

Features of Adobe Experience Manager AEM 6.4

Top Features of Adobe Experience Manager AEM 6.4

• Addition of Style System to template authoring
• Addition of Short Codes in AEM 6.4
• Experience Fragments Building Blocks & Experience Fragments integration with Adobe Target
• Structured Content Fragments and Summarization
• Touch UI Workflow Editor
• Assets Enhancements

Metadata Export and Import –> Import and export metadata in simple csv format.

Cascading Metadata Rules     –> Apply rules on metadata schema to hide/show property based on conditions.

Asset Reports                           –> Generate a detailed report based on selected criteria.

• AEM Smart Tags and Smart Translation Search :-

AEM smart tags and smart translation search are two features that has been introduced in AEM 6.4. 

Smart Tags:- 

               Using Smart Tag content authors can now train their AEM instance to understand the type of asset being uploaded and tag them correctly automatically. 

Smart Translation Search:-  

               AEM Smart Translation Search allows users to perform searches for content in AEM using using non-English terms, it enables cross-language search and discovery automatically across AEM content, both Assets and Pages and supports more than 50 languages. 

Importing an xml file in to AEM

You can write a Java service class (i.e. OSGI service) which reads your XML file data and using JCR API's  it can create corresponding nodes (with data & properties) inside CRX repository and persist it. 

Once u have all the data u can read this anywhere on AEM pages (i.e.websites, widgets etc).  Use  XSLT to convert from one xml format to other.

Below is sample automated imported class for importing xml and convert to AEM xml file.

===================================================================
@Service(value = Importer.class)
@Component
@Property(name = "importer.scheme", value = "importedData", propertyPrivate =true)

public class DataImporter implements Importer {
private final String SOURCE_URL = "http://someserver/data.xml";
private static final Logger LOGGER = LoggerFactory.getLogger(DealerDataImporter.class);

@Override
public void importData(String s, String s2, Resource resource) throws ImportException {

    try {

        URL url = new URL(SOURCE_URL);
        URLConnection connection = url.openConnection();
        Document doc = parseXML(connection.getInputStream());
        NodeList Nodes = doc.getElementsByTagName("retailer");
        for (int i = 0; i < Nodes.getLength(); i++) {
            Element element = (Element) Nodes.item(i);
            String id = element.getElementsByTagName("id").item(0).getTextContent();
            String name = element.getElementsByTagName("display_name").item(0).getTextContent();
            String about = element.getElementsByTagName("about").item(0).getTextContent();
            writeToRepository(id, name, about, resource);
        }
    } catch (Exception e) {
        e.printStackTrace();
    }
}

private void writeToRepository(String id, String name, String about, Resource resource) {

    try {
        javax.jcr.Node parentNode = resource.adaptTo(javax.jcr.Node.class);

        //Say we want this to be a Page node (use NameConstants.NT_PAGE = cq:Page)
        //all of node types can be created this way
        javax.jcr.Node pageNode = JcrUtil.createPath(parentNode.getPath() + "/" + name, NameConstants.NT_PAGE, parentNode.getSession());
        //Page nodes need jcr:content node to hold all teh relevant properties (NameConstants.NN_CONTENT = "jcr:content")
        javax.jcr.Node contentNode = JcrUtil.createPath(pageNode.getPath() + "/" + NameConstants.NN_CONTENT, "cq:PageContent", parentNode.getSession());
        //set some properties
        contentNode.setProperty("about", about);
        contentNode.setProperty("id", id);
        //save session
        parentNode.getSession().save();

    } catch (Exception e1) {
        e1.printStackTrace();
    } 
}
private Document parseXML(InputStream stream){...}

}
=================================================================
After above code execution follow below steps for further processing :

* In CRXde under /etc/importers/polling add new node of type sling:Folder and add some properties:

a) target [String] this is a path to a repository resource this resource will be parsed to your Importer Class.

b) source [String] this is if you're looking to import from multiple xml files IMPORTANT the value needs to start importedData (as a property in the class that is where they link) followed by :SOME_VALIABLE this is a s2 varaible in the above example.

c) interval[Long] how often to run Importer

d) add an mixin jcr:mixinTypes type cq:PollConfig

Thats it.

Migration of content fromTeam Site to CQ5/ AEM

Before we migrate need to consider below points.

• How to migrate existing Team site content store to AEM.
• Around "n" assets( which includes images, docs, pdf, video and audio files).How we can upload assets from Team site to AEM with metadata.
• Migrate existing users & groups to AEM.
• Implement custom login authentication.

Refer below link for more information :  https://forums.adobe.com/thread/2328370

Another  best approach for migration is use of XML files conversion.

Follow below url for more information :
http://cq5aemtutorials.blogspot.com/2018/02/importing-xml-file-in-to-aem.html

Workflow Purging for AEM Performance optimization

Workflow Purging:

Every time a new workflow instance gets created when we launch a workflow (either of asset upload, asset publishing, etc.)

• Once the workflow completes (successful or aborted or terminated), it’s archived and never gets deleted.

• Workflow purging needs to be done to clean up archived workflow instances.

• Purging can be done based on 3 categories of workflows

• Workflow model

• Completion status

• Age of the workflow instance

• To manually purge workflows, execute the operation purgeCompleted on the mbean com.adobe.granite.workflow.

Use “Adobe granite workflow purge configuration” at OSGI configuration to configure automatic workflow purging.

In Older versions of Adobe CQ5/AEM to purge old completed workflows, we need to either write a custom job or need to install a package provided by Adobe/Day Care. But in AEM 5.6.1 onwards we have this functionality built-in.

Workflow instances are stored as nodes inside AEM/CQ.  For long running instances where website users or automated jobs can run workflows, this will quickly add the large amounts of content to repository. 

This leads to slowness of AEM server and also grows disk space.

Ø  ENABLING WORKFLOW PURGE SCHEDULER

It’s not pre-configured feature in AEM. We need to enable AEM to automatically purge workflows.

We can create two configurations of the service to purge workflow instances that satisfy different criteria’s.

ü  First configuration:  purges instances of a particular workflow model which are running for longer duration than expected.

ü  Second configuration: purges all completed workflows after a certain number of days to minimize the size of repository

·         Create a Workflow Purge Scheduler instance.  The Workflow Purge Scheduler is a Sling Service Factory, configuration added to service PID:  com.adobe.granite.workflow.purge.Scheduler

·         As service is a factory service, the name of the sling:OsgiConfig node requires identifier suffix like:   com.adobe.granite.workflow.purge.Scheduler-myidentifier

Add below properties on the node:

Property Name	OSGi Property Name	Description
Job Name	scheduledpurge.name	A descriptive name for the scheduled purge.
Workflow Status	scheduledpurge.workflowStatus	The status of the workflow instances to purge. The following values are valid: ·                     COMPLETED: Completed workflow instances are purged. ·                     RUNNING: Running workflow instances are purged.
Models To Purge	scheduledpurge.modelIds	The ID of the workflow models to purge. The ID is the path to the model node, for example /etc/workflow/models/dam/update_asset/ jcr:content/model. Specify no value to purge instances of all workflow models. To specify multiple models, click the + button in the Web Console.
Workflow Age	scheduledpurge.daysold	The age of the workflow instances to purge,  in days

Once we deploy this file as a content package, we should see configuration show up under the Workflow Purge Scheduler in the OSGi console like below & it will purge workflows which are older than the specified number of days mentioned in configuration.

Ø  Configure Workflow Purge Scheduler in a Package

Deploying a configuration as a part of package deployment process is also one of best approach.  Through the Apache Sling's OSGi Configurations, we can do this with below simple XML file.

·         Create a XML file under a path Eg:  /apps/[my-app]/config

·         Set name of the file to: com.adobe.granite.workflow.purge.Scheduler.config.[some-arbitrary-id].xml

Add below content to XML file and replace values with respective configuration details:

<?xml version="1.0" encoding="UTF-8"?>

<jcr:root xmlns:sling="http://sling.apache.org/jcr/sling/1.0"  xmlns:jcr=http://www.jcp.org/jcr/1.0

jcr:primaryType="sling:OsgiConfig"

scheduledpurge.name="Purge All Completed Workflows"

scheduledpurge.modelIds="[]"

scheduledpurge.workflowStatus="COMPLETED"

scheduledpurge.cron="0 0 * * * ?"

scheduledpurge.daysold="30" />

Once we click save, the scheduler schedules CRON schedule and will purge workflows older than the specified number of days within configuration we specified.

We can also use JMX console to purge repository of completed workflows.

If list of archived workflows grows too large, purge those that are of certain age limit this speeds up page loading.

Use below link for workflow purge scheduler in AEM6.0, we can use AEM built-in feature accessible via JMX and configurable Scheduler http://hostName:portNo/system/console/jmx/com.adobe.granite.workflow%3Atype%3DMaintenance

By mentioning No.of Days of workflow to purge & few more PARAM’s.

For More Info:

http://labs.6dglobal.com/blog/2013-10-03/new-aem-561-workflow-purge-scheduler/

https://docs.adobe.com/content/docs/en/cq/5-6-1/workflows/wf-admin.html#Regular%20Purging%20of%20Workflow%20Instances

TAR Compaction for AEM Performance optimization

TARMK Compaction

We require this when we see growth in TarMk files in the repository. As we all know that tar file data is never overwritten and they always do append the data. So the disk usage increases even when only updating existing data. To avoid such increase in repository,

AEM provides this Tar Compaction. This mechanism will reclaim disk space by removing obsolete data from the repository.

We have two kinds of Tar Compactions.

➢     Online Compaction (AEM 6.2 Doesn’t recommend it anymore)

➢     Offline Compaction

Even before jumping on to these details, let us discussion a bit on automatic compaction triggered manually.

Revision Clean Up: 

               The automatic compaction can be triggered manually in the Operations Dashboard via a maintenance job called Revision Clean Up.

To Start Revision Clean Up you need to:

❏     Go to AEM Welcome Screen.rm

❏     In the main AEM window, go to Tools → Operations → Dashboard → Maintenance

❏     Or directly browse below.

            http://localhost:4502/libs/granite/operations/content/maintenance.html

❏     You see the screen like below.         

❏     Now click on Daily Maintenance Window

❏     Hover over the Revision Clean Up window and press the start button like below.

    ❏     Click on the “Run” icon as seen on the above screenshot.

❏     The icon will turn orange to indicate Revision Clean Up job is running.

You can stop it at any time by hovering the mouse over the icon and pressing the Stop button.

  

Invoking Revision Garbage Collection via the JMX Console

➔     Open the JMX console as

            http://localhost:4502/system/console/jmx

➔     Click RevisionGarbageCollection MBean

➔     In the next window, click startRevisionGC() and then Invoke to start the Revision Garbage Collection job.

TAR OFFLINE COMPACTION:

NOTE: Never do offline compaction when the repository is up and running. Not even checkpoints should be done while the server is up and running.

The procedure is called offline compaction because the repository needs to be shut down in order to properly run the Oak-run tool.

For faster compaction of the Tar files and situations where normal garbage collection does not work, Adobe provides a manual Tar compaction tool called Oak-run. It can be downloaded at the following location: http://mvnrepository.com/artifact/org.apache.jackrabbit/oak-run/

Note: We use Oak-run jar depending upon the version of our current Oak in the AEM server. So make sure we have the right version of oak-run jar.

The procedure to run the tool is:

➔     Always make sure we have a recent working backup of the AEM instance

➔     Shutdown AEM

➔     Use the tool to find old checkpoints:

java -jar oak-run.jar checkpoints <aem-inst-folder>/crx-quickstart/repository/segmentstore

➔     When you run the above command you should be able to see the references of the nodes like below which we need to clear them before compaction takes place.

      

➔     Now delete the unreferenced checkpoints using below command.

            java -jar oak-run.jar checkpoints <aem-inst-folder>/crx-quickstart/repository/segmentstore rm-unreferenced

➔     Finally run the compaction and wait for it to complete

java -jar oak-run.jar compact <aem-inst-folder>/crx-quickstart/repository/segmentstore

➔     You should be see something like below when you run the above command.

           

➔     It should display all files under segmentstore directory. After a while you should be able to see Cleaning up message like below.

➔     Once it cleans up you should be able to see the below message.

➔     It will display the tarfiles again with less count after compaction.

We can create a log file to track all these changes in the log file using the below configurations in the server where we are running the compaction. 

This is one time activity and we need to use the command below to log the information/errors into the log file.

➔     Create the below config file the same place where we have oak-run-*.jar file

➔     Name it as “logback-compaction.xml” and add the below in the lines in xml file.

           

<configuration>

  <appender name="STDERR" class="ch.qos.logback.core.ConsoleAppender">
    <target>System.err</target>
    <encoder>
      <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>
  <logger name="org.apache.jackrabbit.oak.plugins.segment.Compactor" level="INFO"/>
  <root level="warn">
    <appender-ref ref="STDERR" />
  </root>
</configuration>

➔     Now run the below command to log the error/info into the log file.

nohup java -Dtar.memoryMapped=true -Dupdate.limit=5000000 -Dcompaction-progress-log=1500000 -Dcompress-interval=10000000 -Doffline-compaction=true -Dlogback.configurationFile=logback-compaction.xml -Xmx10g -jar <oak jar file path> compact <aem-installation-path> > tarcompaction.log 2>&1

➔     Now should be able to see the file name “tarcompaction.log” the place where you created the above xml config.

TAR OFFLINE COMPACTION PREREQUISTICS:

·                   How to find correct version of oak-run jar?

                    Go to felix console (/system/console/bundles) –> and search for oak, take version across each oak bundle.

      Find Check Points: 

      ----------------------

·                      java -jar /folderPath/aem/oak-run.jar checkpoints F:/AEM/AEM-4502/crx-quickstart/repository/segmentstore

    Run below command on linux/unix machine as windows does not support -Dtar option

·                 java -Dtar.memoryMapped=true -Xmx8g -jar /folderPath/aem/oak-run-jar/oak-run-1.2.7.jar checkpoints /folderPath/aem/crx-quickstart/repository/segmentstore

         Remove CheckPoints:

--------------------------

·                  java -Dtar.memoryMapped=true -Xmx4g -jar /folderPath/aem/oak-run-jar/oak-run-1.2.7.jar checkpoints /folderPath/aem/crx-quickstart/repository/segmentstore rm-unreferenced

Finally Run Compact:

-------------------------

·               java -Dtar.memoryMapped=true -Xmx8g -jar /folderPath/aem/oak-run-jar/oak-run-1.2.7.jar compact /folderPath/aem/crx-quickstart/repository/segmentstore >> /folderPath/aem/help/logs/compactLog

Running Script File:

·     Go to respective script file in folder path.

·     Running below command will execute all the above 3 commands (i.e. finding checkpoints, deleting checkpoints, compacting) are incorporated with in that script file.

Command:

                ./scriptFileName

ONLINE COMPACTION:

                   For situations where the AEM cannot be shut down for maintenance, compaction can also be performed while the instance is running. This is called Online Compaction.

You can configure Online Compaction by doing the following:

➔     Go to the folder where AEM is installed, then browse to crx-quickstart\install

➔     Open the org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.config file.

➔     If that doesn’t exist create one and add the following line to the configuration file

                        repository.home=${repository.home}/segmentstore

            tarmk.size=256

            pauseCompaction=false

➔     Restart AEM

To verify the configuration has taken place, check here.

➔  Go to the JMX console by pointing your browser to

        http://localhost:4502/system/console/jmx

➔  Search for CompactionStrategy and click the MBean that shows up in the search.

➔  Next, verify that the value for PausedCompaction is set to false. This confirms that online compaction is set to run.

➔  Next, verify if Online Compaction is running properly. You can do this by first going to the Operations Dashboard and checking what is the time interval configured for the Daily Maintenance Window. By default, it is scheduled to run between 2 and 5 AM.

➔  Now, inspect the error.log file for events logged during the time of the daily maintenance window to see if online compaction ran correctly.

➔  Example Log:

   [TarMK compaction thread [/author/crx-quickstart/repository/segmentstore], active since Thu Mar 19 02:00:10 EDT 2015, previous max duration 1369831ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction started

19.03.2015 02:00:30.441 *INFO* [pool-9-thread-2] com.adobe.granite.taskmanagement.impl.jcr.TaskArchiveService archiving tasks at: 'Thu Mar 19 02:00:30 EDT 2015'

19.03.2015 02:01:01.699 *INFO* [TarMK compaction thread [/author/crx-quickstart/repository/segmentstore], active since Thu Mar 19 02:00:10 EDT 2015, previous max duration 1369831ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore Estimated compaction in 51.47 s, gain is 69% (1018859520/3343598080) or (1.0 GB/3.3 GB), so running compaction

➔     Log to make sure online compaction is completed.

           

[TarMK compaction thread [/author/crx-quickstart/repository/segmentstore], active since Thu Mar 19 02:00:10 EDT 2015, previous max duration 1369831ms] org.apache.jackrabbit.oak.plugins.segment.file.FileStore TarMK compaction completed in 1310939ms

How to automate the offline compaction process.

Offline Tar Compaction is still the Adobe recommended way of compacting Oak.

Below is the script which automates entire process.

For above process download a version of Oak Run that matches your repository version.

Steps to follow:

1. Shutdown AEM

1. Find Old Checkpoints

1. Remove Unreferenced Checkpoints

1. Compact Oak (using compact keyword in command).

1. Restart AEM

SCRIPT:

#!/bin/bash

todayDate="$(date +'%d-%m-%Y')"

logfile="compact-$ todayDate.log"

installfolder="/data/aem"

aemfolder="$installfolder/crx-quickstart"

oakrun="$installfolder/help/oak-run-1.0.18.jar"

## Shutdown AEM

printf "Shutting down AEM.\n"

$aemfolder/bin/stop

todayDate ="$(date)"

echo "AEM Shutdown at: $ todayDate " >> $installfolder/help/logs/$logfile

## Find old checkpoints

printf "Finding old checkpoints.\n"

java -Dtar.memoryMapped=true -Xms8g -Xmx8g -jar $oakrun checkpoints $aemfolder/repository/segmentstore >> $installfolder/help/logs/$logfile

## Delete unreferenced checkpoints

printf "Deleting unreferenced checkpoints.\n"

java -Dtar.memoryMapped=true -Xms8g -Xmx8g -jar $oakrun checkpoints $aemfolder/repository/segmentstore rm-unreferenced >> $installfolder/help/logs/$logfile

## Run compaction

printf "Running compaction. This may take a while.\n"

java -Dtar.memoryMapped=true -Xms8g -Xmx8g -jar $oakrun compact $aemfolder/repository/segmentstore >> $installfolder/help/logs/$logfile

## Report Completed

printf "Compaction complete. Please check the log at:\n"

printf "$installfolder/help/logs/$logfile\n"

## Start AEM back up

todayDate ="$(date)"

printf "Starting up AEM.\n"

$aemfolder/bin/start

echo "AEM Startup at: $ todayDate " >> $installfolder/help/logs/$logfile

        http://experiencemanaged.com/posts/scripting-aem-oak-compaction.html

        https://jackrabbit.apache.org/oak/docs/command_line.html