Xpand2

Xpand2

The Xpand language is used in templates to control the output generation. This documentation describes the general syntax and semantics of the Xpand language.

Typing the guillemets (« and ») used in the templates is supported by the Eclipse editor, which provides keyboard shortcuts with Ctrl + < and Ctrl + > .

Templates are stored in files with the extension .xpt . Template files must reside on the Java classpath of the generator process.

Almost all characters used in the standard syntax are part of ASCII and should therefore be available in any encoding . The only limitation are the tag brackets ( guillemets ), for which the characters "«" (Unicode 00AB) and "»" (Unicode 00BB) are used. So for reading templates, an encoding should be used that supports these characters (e.g. ISO-8859-1 or UTF-8).

Names of properties, templates, namespaces etc. must only contain letters, numbers and underscores.

Here is a first example of a template:

«IMPORT meta::model»
«EXTENSION my::ExtensionFile»

«DEFINE javaClass FOR Entity»
   «FILE fileName()»
      package «javaPackage()»;

      public class «name» {
         // implementation
      }
   «ENDFILE»
«ENDDEFINE»

A template file consists of any number of IMPORT statements, followed by any number of EXTENSION statements, followed by one or more DEFINE blocks (called definitions).

The central concept of Xpand is the DEFINE block, also called a template. This is the smallest identifiable unit in a template file. The tag consists of a name, an optional comma-separated parameter list, as well as the name of the metamodel class for which the template is defined.

«DEFINE templateName(formalParameterList) FOR MetaClass»
   a sequence of statements
«ENDDEFINE»

To some extend, templates can be seen as special methods of the metaclass. There is always an implicit this parameter which can be used to address the "underlying" model element; in our example above, this model element is of type "MetaClass".

As in Java, a formal parameter list entry consists of the type followed by the name of that parameter.

The body of a template can contain a sequence of other statements including any text.

A full parametric polymorphism is available for templates. If there are two templates with the same name that are defined for two metaclasses which inherit from the same superclass, Xpand will use the corresponding subclass template, in case the template is called for the superclass. Vice versa, the template of the superclass would be used in case a subclass template is not available. Note that this not only works for the target type, but for all parameters. Technically, the target type is handled as the first parameter.

So, let us assume you have the following metamodel:


Assume further, you would have a model which contains a collection of A, B and C instances in the property listOfAs. Then, you can write the following template:

«DEFINE someOtherDefine FOR SomeMetaClass»
   «EXPAND implClass FOREACH listOfAs»
«ENDDEFINE»

«DEFINE implClass FOR A»
   // this is the code generated for the superclass A
«ENDDEFINE»

«DEFINE implClass FOR B»
   // this is the code generated for the subclass B
«ENDDEFINE»

«DEFINE implClass FOR C»
   // this is the code generated for the subclass C
«ENDDEFINE»

So for each B in the list, the template defined for B is executed, for each C in the collection the template defined for C is invoked, and for all others (which are then instances of A) the default template is executed.

The EXPAND statement "expands" another DEFINE block (in a separate variable context), inserts its output at the current location and continues with the next statement. This is similar in concept to a subroutine call.

«EXPAND definitionName [(parameterList)]
   [FOR expression | FOREACH expression [SEPARATOR expression] ] [ONFILECLOSE]»

The various alternative syntaxes are explained below.

Appending the ONFILECLOSE statement defers evaluation of the expanded definition until the current file is closed with ENDFILE. This is of use when the state required to create the text is collected during the evaluation of the processed definition.

«FILE ...»
...
«EXPAND LazyEvaluatedDefinition FOREACH myCollection ONFILECLOSE»
...
«ENDFILE» «REM»Now 'LazyEvaluatedDefinition' is called«ENDFILE»

A typical example for usage of the ONFILECLOSE statement is when you want to create a list of imports in a Java class, but the types that are used should be added when they are used in the templates later.

The state, usually a collection, that is used for the lazy expanded evaluation must be valid until the file is closed. This can be achieved in two ways:

Protected Regions are used to mark sections in the generated code that shall not be overridden again by the subsequent generator run. These sections typically contain manually written code.

«PROTECT CSTART expression CEND expression ID expression (DISABLE)?»
   a sequence of statements
«ENDPROTECT»

The values of CSTART and CEND expressions are used to enclose the protected regions marker in the output. They should build valid comment beginning and comment end strings corresponding to the generated target language (e.g. "/*" and "*/" for Java).

The following is an example for Java:

«PROTECT CSTART "/*" CEND "*/" ID ElementsUniqueID»
   here goes some content
«ENDPROTECT»

The ID is set by the ID expression and must be globally unique (at least for one complete pass of the generator). To assure this these IDs are usually concatenated. Some model types (e.g. UML2 models) contain identifiers that could be used, which can be read using the xmlId() function from stdlib.

Generated target code looks like this:

public class Person {
/*PROTECTED REGION ID(Person) ENABLED START*/
   This protected region is enabled, therefore the contents will
   always be preserved. If you want to get the default contents
   from the template you must remove the ENABLED keyword (or even
   remove the whole file :-))
/*PROTECTED REGION END*/
}

Protected regions are generated in enabled state by default. Unless you manually disable them, by removing the ENABLED keyword, they will always be preserved.

If you want the generator to generate disabled protected regions, you need to add the DISABLE keyword inside the declaration:

«PROTECT CSTART '/*' CEND '*/' ID this.name DISABLE»

The produced target code won't contain the ENABLED flag then. In this case ENABLED has to be added to the target region to activate the protected region. Disabling protected regions by default has the advantage that the protected region default content in the template can be changed and all not yet activated regions would contain the changed code after regeneration.

Using the workflow engine it is now possible to package ( e.g. zip) a written generator and deliver it as a kind of black box (this is often called a cartridge). If you want to use such a generator but need to change some small generation stuff, you can make use of the AROUND aspects.

«AROUND qualifiedDefinitionName(parameterList)? FOR type»
   a sequence of statements
«ENDAROUND» 

AROUND lets you add templates in an non-invasive way (you do not need to touch the generator templates). Because aspects are invasive, a template file containing AROUND aspects must be wrapped by configuration (see next section).

AOP is basically about weaving code into different points inside the call graph of a software module. Such points are called Join Points . In Xpand , there is only one join point so far: a call to a definition.

You specify on which join points the contributed code should be executed by specifying something like a 'query' on all available join points. Such a query is called a point cut .

«AROUND [pointcut]»
   do stuff
«ENDAROUND»

A point cut consists of a fully qualified name, parameter types and the target type.

This section describes the workflow component that is provided to perform the code generation, i.e. run the templates. You should have a basic idea of how the workflow engine works. A simple generator component configuration could look as follows:

<component class="org.eclipse.xpand2.Generator">
   <fileEncoding value="ISO-8859-1"/>
   <metaModel class="org.eclipse.xtend.typesystem.emf.EmfMetaModel">
       <metaModelPackage value="org.eclipse.emf.ecore.EcorePackage"/>
   </metaModel>
   <expand value="somenamespace::example::Java::all FOR myModel"/>

   <!-- aop configuration -->
   <advices value='somenamespace::example::Advices1, example::Advices2'/>

   <!--  output configuration -->
   <outlet path='main/src-gen'>
     <postprocessor class="org.eclipse.xpand2.output.JavaBeautifier"/>
     <postprocessor class="org.eclipse.xtend.typesystem.xsd.XMLBeautifier"/>
   </outlet>
   <outlet name='TO_SRC' path='main/src' overwrite='false'>
     <postprocessor class="org.eclipse.xpand2.output.JavaBeautifier"/>
     <postprocessor class="org.eclipse.xtend.typesystem.xsd.XMLBeautifier"/>
   </outlet>

   <!-- optional: protected regions configuration -->
   <prSrcPaths value="main/src"/>
   <prDefaultExcludes value="false"/>
   <prExcludes value="*.xml"/>
</component>

Now, let us go through the different properties one by one.

For Xpand it is important to have the file encoding in mind because of the guillemet characters « » used to delimit keywords and property access. The fileEncoding property specifies the file encoding to use for reading the templates, reading the protected regions and writing the generated files. This property defaults to the default file encoding of your JVM.

In a team that uses different operating systems or locales it is a good idea to set the file encoding fixed for the Xpand project and share the settings. Typical encodings used are UTF-8 or ISO-8859-1, but any encoding having guillemet brackets is fine also.[8]

An false encoding can result in an error message of the generator during runtime:

1108 ERROR WorkflowRunner     - [ERROR]: no viable alternative at input 'Â' on line 1

In this case you have to configure the input encoding. A ResourceManager is used to set the input encoding. Use the fileEncoding property of the ResourceManager inside the generator component to configure the encoding of templates and extensions.

Example for MWE :

<component class="org.eclipse.xpand2.Generator">
 <metaModel idRef="mm_emf"/>
 <expand
  value="template::Template::main FOR model" />
 <outlet path="${src-gen}" >
  <postprocessor class="org.eclipse.xpand2.output.JavaBeautifier" />
 </outlet>
 <resourceManager class ="org.eclipse.xtend.expression.ResourceManagerDefaultImpl">
  <fileEncoding value="ISO-8859-1"/>
 </resourceManager>
</component>

Example for MWE2 :

component = org.eclipse.xpand2.Generator {
    metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {}
    expand = "templates::Template::main FOREACH model"
    outlet = {
        path = targetDir
    }
    resourceManager = org.eclipse.xtend.expression.ResourceManagerDefaultImpl {
        fileEncoding = "ISO-8859-1"
    }
}

The section Output configuration describes how to configure the encoding of the generated files.

The second mandatory configuration is the specification of so called outlets (a concept borrowed from AndroMDA). Outlets are responsible for writing the generated files to disk.

Example MWE :

<component class="org.eclipse.xpand2.Generator">
   ...
   <outlet path='main/src-gen'/>
   <outlet name='TO_SRC' path='main/src' overwrite='false'>
      <fileEncoding value='ISO-8859-1'/>
   </outlet>
   <fileEncoding value='ISO-8859-1'/>
   ...
</component>

Example MWE2 :

component = org.eclipse.xpand2.Generator {
    metaModel = org.eclipse.xtend.typesystem.emf.EmfRegistryMetaModel {}
    expand = "templates::Template::main FOREACH model"
    outlet = {path = 'main/src-gen'} 
    outlet = {
        name='TO_SRC' 
        path='main/src' 
        overwrite= false
        fileEncoding = 'ISO-8859-1'
    }
    fileEncoding = 'ISO-8859-1'
    ...
}

In the examples there are two outlets configured. The first one has no name and is therefore handled as the default outlet. Default outlets are triggered by omitting an outlet name:

«FILE 'test/note.txt'»
# this goes to the default outlet
«ENDFILE»

The configured base path is 'main/src-gen', so the file from above would go to 'main/src-gen/test/note.txt'.

The second outlet has a name ('TO_SRC') specified. Additionally the flag overwrite is set to false (defaults to true). The following Xpand fragment

«FILE 'test/note.txt' TO_SRC»
# this goes to the TO_SRC outlet
«ENDFILE»

would cause the generator to write the contents to 'main/src/test/note.txt' if the file does not already exist (the overwrite flag).

Another option called append (defaults to false) causes the generator to append the generated text to an existing file. If overwrite is set to false this flag has no effect.

The encoding of the generated files can be configured at two different levels. A file encoding can be defined for the complete generator component. Therefore the fileEncoding property inside the component definition has to be used (see the examples above). You can also define a file encoding at outlet level. Therefore the fileEncoding property inside the outlet definition has to be used.

Beautifying the generated code is a good idea. It is very important that generated code looks good, because developers should be able to understand it. On the other hand template files should look good, too. It is thus best practice to write nice looking template files and not to care how the generated code looks - and then you run a beautifier over the generated code to fix that problem. Of course, if a beautifier is not available, or if white space has syntactical meaning (as in Python), you would have to write your templates with that in mind (using the minus character before closing brackets as described in a preceding section).

The Xpand workflow component can be configured with multiple beautifiers:

<outlet ...>
   <postprocessor class="org.eclipse.xpand2.output.JavaBeautifier"/>
   <postprocessor class="org.eclipse.xtend.typesystem.xsd.XMLBeautifier"/>
</outlet>

These are the two beautifiers delivered with Xpand . If you want to use your own beautifier, you would just need to implement the PostProcessor Java interface:

package org.eclipse.xpand2.output;

public interface PostProcessor {
   public void beforeWriteAndClose(FileHandle handle);
   public void afterClose(FileHandle handle);
}

The beforeWriteAndClose method is called for each ENDFILE statement.

PostProcessors can also be used for othermeans than formatting, like line counting.

The Xpand engine will generate code for each processed FILE statement. This implies that files are written that might not have changed to the previous generator run. Normally it does not matter that files are rewritten. There are at least two good reasons when it is better to avoid rewriting of files:

  1. The generated source code will be checked in. In general it is not the recommended way to go to check in generated code, but sometimes you will have to. Especially with CVS there is the problem that rewritten files are recognized as modified, even if they haven't changed. So the problem arises that identical files get checked in again and again (or you revert it manually). When working in teams the problem even becomes worse, since team members will have conflicts when checking in.

  2. When it can be predicted that the generator won't produce different content before a file is even about to be created by a FILE statement then this can boost performance. Of course it is not trivial to predict that a specific file won't result in different content before it is even created. This requires information from a prior generator run and evaluation against the current model to process. Usually a diff model would be used as input for the decision.

Case 1) will prevent file writing after a FILE statement has been evaluated, case 2) will prevent creating a file at all.

To achieve this it is possible to add Veto Strategies to the generator, which are implementations of interface org.eclipse.xpand2.output.VetoStrategy or org.eclipse.xpand2.output.VetoStrategy2. Use VetoStrategy2 if you implement your own.

VetoStrategy2 declares two methods:

  • boolean hasVetoBeforeOpen (FileHandle)

    This method will be called before a file is being opened and generated. Return true to suppress the file creation.

  • boolean hasVeto (FileHandle)

    This method will be called after a file has been produced and after all configured PostProcessors have been invoked. Return true to suppress writing the file.

Veto Strategies are configured per Outlet. It is possible to add multiple stratgy instances to each Outlet.

  <component id="generator" class="org.eclipse.xpand2.Generator" skipOnErrors="true">
    <metaModel class="org.eclipse.xtend.typesystem.uml2.UML2MetaModel"/>
    <expand value="templates::Root::Root FOR model"/>
    <fileEncoding value="ISO-8859-1"/>
      <outlet path="src-gen">
         <postprocessor class="org.eclipse.xpand2.output.JavaBeautifier"/>
         
<vetoStrategy class="org.eclipse.xpand2.output.NoChangesVetoStrategy"/>

      </outlet>
  </component>

One VetoStrategy is already provided. The org.eclipse.xpand2.output.NoChangesVetoStrategy is a simple implementation that will compare the produced output, after it has been postprocessed, with the target file. If the content is identical the strategy vetoes the file writing. This strategy is effective, but has two severe drawbacks:

  1. The file has been created at least in memory before. This consumes time and memory. If applying code formatting this usually implies that the file is temporarily written.

  2. The existing file must be read into memory. This also costs time and memory.

Much better would be to even prevent the creation of files by having a valid implementation for the hasVetoBeforeOpen() method. Providing an implementation that predicts that files do not have to be created requires domain knowledge, thus a standard implementation is not available.

The number of skipped files will be reported by the Generator component like this:

2192 INFO  - Generator(generator): generating <...>
3792 INFO  - 
Skipped writing of 2 files to outlet
 [default](src-gen)

This example shows how to use aspect-oriented programming techniques in Xpand templates. It is applicable to EMF based and Classic systems. However, we explain the idea based on the emfExample . Hence you should read that before.

There are many circumstances when template-AOP is useful. Here are two examples:

Scenario 1: Assume you have a nice generator that generates certain artifacts. The generator (or cartridge) might be a third party product, delivered in a single JAR file. Still you might want to adapt certain aspects of the generation process without modifying the original generator .

Scenario 2: You are building a family of generators that can generate variations of the generate code, e.g. Implementations for different embedded platforms. In such a scenario, you need to be able to express those differences (variabilities) sensibly without creating a non-understandable chaos of if statements in the templates.

To illustrate the idea of extending a generator without "touching" it, let us create a new project called org.eclipse.demo.emf.datamodel.generator-aop. The idea is that it will "extend" the original org.eclipse.demo.emf.datamodel.generator project introduced in the emfExample . So this new projects needs to have a project dependency to the former one.

An AOP system always needs to define a join point model; this is, you have to define, at which locations of a (template) program you can add additional (template) code. In Xpand , the join points are simply templates (i.e. DEFINE .. ENDDEFINE ) blocks. An "aspect template" can be declared AROUND previously existing templates. If you take a look at the org.eclipse.demo.emf.datamodel.generator source folder of the project, you can find the Root.xpt template file. Inside, you can find a template called Impl that generates the implementation of the JavaBean.

«DEFINE Entity FOR data::Entity»
   «FILE baseClassFileName() »
      // generated at «timestamp()»
      public abstract class «baseClassName()» {
         «EXPAND Impl»
      }
   «ENDFILE»
«ENDDEFINE»

«DEFINE Impl FOR data::Entity»
   «EXPAND GettersAndSetters»
«ENDDEFINE»

«DEFINE Impl FOR data::PersistentEntity»
   «EXPAND GettersAndSetters»
    public void save() {

   }
«ENDDEFINE»

What we now want to accomplish is this: Whenever the Impl template is executed, we want to run an additional template that generates additional code (for example, some kind of meta information for a given framework. The specific code at this place is not important for the example here).

So, in our new project, we define the following template file:

«AROUND Impl FOR data::Entity»
   «FOREACH attribute AS a»
      public static final AttrInfo «a.name»Info = new AttrInfo(
         "«a.name»", «a.type».class );
   «ENDFOREACH»
   «targetDef.proceed()»
«ENDAROUND»

So, this new template wraps around the existing template called Impl It first generates additional code and then forwards the execution to the original template using targetDef.proceed(). So, in effect, this is a BEFORE advice. Moving the proceed statement to the beginning makes it an AFTER advice, omitting it, makes it an override.

In general, the syntax for the AROUND construct is as follows:

«AROUND fullyQualifiedDefinitionNameWithWildcards
      (Paramlist (*)?) FOR TypeName»
   do Stuff
«ENDAROUND»

Here are some examples:

«AROUND *(*) FOR Object»

matches all templates

«AROUND *define(*) FOR Object»

matches all templates with define at the end of its name and any number of parameters

«AROUND org::eclipse::xpand2::* FOR Entity»

matches all templates with namespace org::eclipse::xpand2:: that do not have any parameters and whose type is Entity or a subclass

«AROUND *(String s) FOR Object»

matches all templates that have exactly one String parameter

«AROUND *(String s,*) FOR Object»

matches all templates that have at least one String parameter

«AROUND my::Template::definition(String s) FOR Entity»

matches exactly this single definition

Inside an AROUND, there is the variable targetDef, which has the type xpand2::Definition. On this variable, you can call proceed, and also query a number of other things:

«AROUND my::Template::definition(String s) FOR String»
   log('invoking '+«targetDef.name»+' with '+this)
   «targetDef.proceed()»
«ENDAROUND»


[8] On Mac OSX the default encoding is MacRoman, which is not a good choice, since other operating systems are not aware of this encoding. It is recommended to set the encoding to some more common encoding, e.g. UTF-8, maybe even for the whole workspace.