2016-07 Open Source for Adobe Experience Manager

https://github.com/TWCable/

Time Warner Cable was acquired by Charter Communications in May 2016

Why open-source what we’ve so worked hard on?

Hoping to encourage the community to innovate on higher-value things by getting this stuff out of the way
Encouraging others to help make the tooling better
Show a glimpse of some of the cool work we’re doing and how much we value engineers doing original and interesting projects

Agenda

Jackalope
Gradle Plugins
Grabbit

Simple test doubles for the JCR/Sling/CQ

Focus on simplicity, speed and clarity

Basic Jackalope Examples

import static com.twcable.jackalope.JCRBuilder.node as n 
import static com.twcable.jackalope.JCRBuilder.property as p 
import static com.twcable.jackalope.JCRBuilder.repository

 def repository = repository( 
  n("content",
    n("test1",
      n("callingrates",
        n("intl-direct-dial",
          p("sling:resourceType",
            "admin/components/content/callingratetable"),
          n("france",
            p("sling:resourceType",
              "admin/components/content/callingrate"),
            p("additional-minute-rate", "0.60"))))))).build()

def resolver = new SimpleResourceResolverFactory(repository).
                     administrativeResourceResolver 
def resource = resolver.getResource("/content/test1/" +
                  "callingrates/intl-direct-dial")

import static com.twcable.jackalope.JCRBuilder.node
 import static com.twcable.jackalope.JCRQueryBuilder.query
 import static com.twcable.jackalope.JCRQueryBuilder.result

  given:
 def node = node("result").build()
 def session = node.session  

JCRQueryBuilder.queryManager(session,
                             query("SELECT ...",
                                   JCR_SQL2, result(node))
).build()

 when:
 def queryResult = session.workspace.queryManager.
                    createQuery("SELECT ...", JCR_SQL2).execute() 

 then:
 queryResult.nodes.first() == node

Primary Jackalope Alternatives

CITYTECH’s Prosper
- Very Groovy and dynamic DSL
Sling JCR Mocks
- Essentially the “normal” API
- Verbose and does not “show” the structure

CQ/AEM Gradle Plugins

Bridges the gap with Maven’s CQ support

buildscript {
    repositories {
        maven {
            url "http://dl.bintray.com/twcable/aem"
        }
        dependencies {
            classpath "com.twcable.gradle:gradle-plugin-scr:1.1.0"
            classpath "com.twcable.gradle:gradle-plugin-cq-bundle:2.1.0"
            classpath "com.twcable.gradle:gradle-plugin-cq-package:2.1.0"
        }
    }
}

apply plugin: "com.twcable.scr"
apply plugin: "com.twcable.cq-bundle"
apply plugin: "com.twcable.cq-package"

gradle-plugin-scr

Adds the processScrAnnotations task to processes the @SCR annotations and create the appropriate OSGi metadata for OSGi Declarative Services

gradle-plugin-cq-bundle

uploadBundle	Uploads the bundle to the CQ server. The task will fail if the bundle fails to start.
startBundle	Start the bundle on the servers. This task will fail if the bundle does not exist on the server.
stopBundle	Stop the bundle on the servers. This task will *not* fail if the bundle does not exist on the server.
removeBundle	Uninstalls and deletes the bundle on the servers. This task will *not* fail if the bundle does not exist on the server.
refreshAllBundles	Tells CQ to refresh all the bundles. This gets added to the top-level project and generally a good idea to run after any of the other tasks.

gradle-plugin-cq-package

createPackage	This will create vault package, adding special features to make it easier to work with for the specifics that CQ wants.
uploadPackage	Upload the package to all the servers defined by the `slingServers` configuration. If the package already exists (even as a "-SNAPSHOT"), you probably want to uninstall and remove it before uploading a new one. See AEM 6.1 Packages and Bundles - Installing and Uninstalling Behavior for much more information on why. For that reason it depends on `removePackage` by default.

installPackage	Installs the CQ Package that has been uploaded.
uninstallPackage	Uninstalls the CQ Package. If the package is not on the server, nothing happens. (i.e., This does not fail if the package is not on the server.)
removePackage	Removes the package from CQ. Does not fail if the package was not on the server to begin with. This depends on `uninstallPackage` so that the old content is not left behind.

validateBundles	Checks all the JARs that are included in this package to make sure they are installed and in an ACTIVE state. Gives a report of any that are not. This task polls using the server settings for retries and max time in the case the bundles are newly installed.
validateRemoteBundles	The same as validateBundles but downloads the package as it exists on the remote server, then extracts it to get the bundles inside it.
uninstallBundles	Downloads the currently installed package .zip file if it exists, compiles a list of bundles based off what is currently installed, then stops and uninstalls each bundles individually.

addBundlesToFilterXml	Adds the bundles to the filter.xml
verifyBundles	Checks all the JARs that are included in this package to make sure they are OSGi compliant, and gives a report of any that are not. Never causes the build to fail.
startInactiveBundles	Asynchronously attempts to start any bundle in a RESOLVED state.

Grabbit

Fast Content Copy for AEM

Background

The Primary Alternatives

Package Manager
vlt rcp (and Mark Adamcin’s “recap”)

Package Manager Pros

it works out the box
handles all the edge-cases
well documented (well, kinda-sorta)
fast

Package Manager Cons

no (realistic) deltas
EXTREMELY space-inefficient
- sheer space
- generates a lot of extra I/O
gets wonky with packages in the GB range

vlt rcp Pros

it works out the box
handles all the edge-cases
space efficient
not too horribly slow when all machines are on the same network

vlt rcp Cons

very sensitive to network latency (terrible on a WAN/cloud)
when something goes wrong, the error messages are pretty bad

What is Grabbit?

Grabbit is focussed on speed, reliability and monitoring

Not intended to replace Packages

Should do the ~90% of keeping content up-to-date

Grabbit Pros

data-transfer is very, very optimized
- network topology matters very little
- depending on the kind of data and network, generally see 200%-1,000% faster than vlt rcp
space efficient
deltas
clear-before-write

Grabbit Cons

“edge cases” are not fully supported
- if you can’t write with the JCR API (e.g., Users/Groups) it may not work
- Vault (Package Manager and vlt rcp) are part of Jackrabbit, so it will always receive the most love for backwards compatibility, changes to implementation details in the JCR, etc.
large (multi-GB) paths require special handling

Installation

Install two packages on both the Client and Server AEM/CQ machines
- Sun-Misc-Fragment-Bundle-1.0.0.zip
- Grabbit.zip

PUT /grabbit/job

# Information for connecting to the source content
serverUsername : '<username>'
serverPassword : '<password>'
serverHost : some.other.server
serverPort : 4502

deltaContent : true # default for all the paths

# A reference to the standard set of workflow configuration ids that
# we want to turn off when working with DAM assets.
damWorkflows: &ourDamWorkflows
  - /etc/workflow/launcher/config/update_asset_mod
  - /etc/workflow/launcher/config/update_asset_create
  - /etc/workflow/launcher/config/dam_xmp_nested_writeback
  - /etc/workflow/launcher/config/dam_xmp_writeback

# Each of the paths to include in the copy
pathConfigurations :
  -
    path : /content/someContent
  -
    path : /content/someOtherContent
    excludePaths: [ someExcludeContent ]
  -
    path : /content/dam/someDamContent
    excludePaths :
      - someContent/someExcludeContent
      - someContent/someOtherExcludeContent
    workflowConfigIds : *ourDamWorkflows

GET /grabbit/job/all.json

GET /grabbit/job/<id>.json

{
    "transactionID": 4364570344328332300,
    "jobExecutionId": 3416428771481128000,
    "jcrNodesWritten": 1,
    "exitStatus": {
        "exitDescription": "",
        "exitCode": "COMPLETED",
        "running": false
    },
    "endTime": "2016-04-25T16:01:46+0000",
    "timeTaken": 4212,
    "path": "/content/campaigns/jcr:content",
    "startTime": "2016-04-25T16:01:42+0000"
}

Google’s Protocol Buffers

Pros

efficient for both space and time
- binary w/ compression
extensible
battle-tested

Google’s Protocol Buffers

Cons

Not meant for large (MB) blobs of data

message Node {
    required string name = 1;
    required Properties properties = 2;
    repeated Node mandatoryChildNode = 3;
}
message Properties {
    repeated Property property = 1;
}
message Property {
    required string name = 1;
    required int32 type = 2;
    optional Value value = 3;
    optional Values values = 4;
}
message Values {
    repeated Value value = 1;
}
message Value {
    optional string stringValue = 1;
    optional bytes bytesValue = 2;
}

Processing the Jobs

<batch:job id="clientJob"
    xmlns="http://www.springframework.org/schema/batch"
    job-repository="clientJobRepository">

    <!-- ... -->

</batch:job>

<batch:decision id="validateJob"
    decider="validJobDecider">

    <next on="VALID" to="clientWorkflowOff" />
    <fail on="INVALID" exit-code="VALIDATION_FAILED" />
</batch:decision>

<batch:step id="clientWorkflowOff"
    next="deleteBeforeWriteDecision">

    <batch:tasklet ref="clientWorkflowOffTasklet"
        transaction-manager="clientTransactionManager"/>
</batch:step>

<batch:decision id="deleteBeforeWriteDecision"
    decider="deleteBeforeWriteDecider">

    <next on="NO" to="startHttpConnection"/>
    <next on="YES" to="deleteBeforeWrite" />
</batch:decision>

<batch:step id="deleteBeforeWrite"
    next="startHttpConnection">

    <batch:tasklet ref="deleteBeforeWriteTasklet"
        transaction-manager="clientTransactionManager"/>
</batch:step>

<batch:step id="startHttpConnection"
    next="clientNamespaceSync">

    <batch:tasklet ref="createHttpConnectionTasklet"
        transaction-manager="clientTransactionManager"/>
</batch:step>

<!-- Writes JCR namespaces streamed from the server-->
<batch:step id="clientNamespaceSync"
    next="clientJcrNodes">

    <batch:tasklet ref="clientNamespaceSyncTasklet"
        transaction-manager="clientTransactionManager"/>
</batch:step>

<batch:step id="clientJcrNodes" next="clientWorkflowOn">
    <batch:tasklet
        transaction-manager="clientTransactionManager">
        <batch:chunk reader="clientProtobufNodesReader"
            writer="clientJcrNodesWriter"
            commit-interval="#{jobParameters[batchSize]}"
            skip-limit="0"/>
        <listeners>
            <listener ref="loggingStepExecutionListener"/>
        </listeners>
    </batch:tasklet>
</batch:step>

<!-- Read ProtoBuf message -->
<bean id="clientProtobufNodesReader"
    class="c.t.g.c.batch.steps.jcrnodes.JcrNodesReader"/>

<!-- Write JCR node -->
<bean id="clientJcrNodesWriter"
    class="c.t.g.c.batch.steps.jcrnodes.JcrNodesWriter"/>

<!-- uses a countdown latch to make sure all other jobs
     that turned off the workflow have finished -->
<batch:step id="clientWorkflowOn">
    <batch:tasklet ref="clientWorkflowOnTasklet"
        transaction-manager="clientTransactionManager"/>
</batch:step>

<!-- job lifecycle setup/cleanup -->
<batch:listeners>
    <listener ref="clientBatchJobListener"/>
</batch:listeners>

2016-07 Open Source for Adobe Experience Manager

Agenda

Simple test doubles for the JCR/Sling/CQ

Basic Jackalope Examples

Primary Jackalope Alternatives

CQ/AEM Gradle Plugins

gradle-plugin-scr

gradle-plugin-cq-bundle

gradle-plugin-cq-package

Grabbit

Fast Content Copy for AEM

Background

The Primary Alternatives

Package Manager Pros

Package Manager Cons

vlt rcp Pros

vlt rcp Cons

What is Grabbit?

Grabbit Pros

Grabbit Cons

Installation

PUT /grabbit/job

Google’s Protocol Buffers

Google’s Protocol Buffers

Processing the Jobs

Q & A