DIM Custom Plugin Reference Guide

DIM Custom Plugin Reference Guide

Overview

The Data Integration Module (DIM) allows Users to create custom Plugins that can directly interface with the Data Integration Service and Projects. This document describes how to setup and use Plugins with the DIM. 

DIM Processing Overview 

The Data Integration Module executes processing instructions from top to bottom as defined in the XML project file. First, it sends each processor an ‘execute’ message in order. After each processor indicates successful execution, it sends a ‘commit’ message to finalize changes. This sequence establishes a transaction for atomic dispatch processing. However, processors are not required to participate in the transaction and may finalize changes directly during the execute phase. 

If any processor fails to execute or commit successfully, the DIM sends a ‘rollback’ message to all processors that completed execution. Because an error may happen during the commit phase, it is possible for a processor to receive a rollback message after it has already received a commit message. Most processors will simply ignore a rollback message received after a commit message.
 

Plugin Details and Requirements 

You can include custom data processing code in DIM execution by writing a .NET 2.0 class implementing the IPlugin2 and IDispatchProcessor interfaces from TransformerSdk.dll. Add your plugin to the processing sequence using a processPlugin element in the project XML file. Your plugin assembly and any other libs it depends on should be copied into the DIM Program Files folder. The default location for the DIM is one of following: 
  1. 32 Bit OS: C:\Program Files\F2B Data Integration Module 
  2. 64 Bit OS: C:\Program Files (x86)\F2B Data Integration Module 
The processPlugin element must give the assembly-qualified name of your plugin class using a class attribute. 

IPlugin2 defines an Initialize method that your plugin should implement to do any startup tasks when the DIM loads the project file. The plugin receives the processPlugin XML element from the project XML file, which can be used for instance configuration. For example, a plugin that creates files may specify the location for new files as a child of processPlugin. The plugin also receives an IPluginHost containing a WaitHandle that gets signaled when the DIM service is shutting down. 

IDispatchProcessor defines methods Execute, Commit, and Rollback. Implement these methods to execute your custom processing code at each stage of processing. 
The Execute method receives dispatch data using the IDispatch interface. This interface contains some information such as the dispatch reference number and sender’s username. It also lists the rendered files included in the dispatch and contains a list of ISentFormData instances providing access to sent form data. Most dispatches will simply contain one sent form. 


DIM Interface Definitions 

The following section outlines the definitions for the Integration and Transformer SDK’s of the DIM Plugin API. 

Field2Base.Integration.Sdk Namespace 
IDispatch Interface 
Accesses dispatch data 
Properties 
int ReferenceNumber { get; } 
Get the reference number assigned to this dispatch by F2B 
string SenderUsername { get; } 
Get the username of the user who sent this dispatch 
IList<ISentFormData> Forms { get; } 
Get form data included in this dispatch 
IList<FileInfo> ContentFiles { get; } 
Get a list of files included in the dispatch 
 
ISentFormData Interface 
Accesses form data 
 
Properties 
XmlDocument Data { get; } 
Direct access to the XML data file. Null if the data file is missing. 
 
Methods 
string GetRegionValue(string pRegionFullName); 
Helper to get a region value 
Returns: Serialized string region value; null if the region doesn't exist or was left blank 
string GetHeaderValue(string pHeaderKey); 
Helper to get a header value 
Returns: Header value; null if the value doesn't exist 
byte[] GetPageImage(int pPageIndex); 
Get image data for a rendered form page using its 0-based index in the form 
Returns: Page image; null if the page doesn't exist or page image data isn't available 
 
IVariableBindings Interface 
Carries execution parameters and variables during form processing. i.e. The min and max date values for restricting processing to a date range. 
Methods 
object Lookup(string pIdentifier); 
Get the value bound to an identifier 
Returns: The value bound to an identifier; null if the identifier is not bound. 
 
Field2Base.Transformer.Sdk Namespace 
IPluginHost interface 
Provides an environment for plugins and form processing in general 
Properties 
WaitHandle StopSignal { get; } 
Signal set when the host is trying to shut down 
 
IPlugin2 interface 
Generic interface for extensions 
Methods 
void Initialize(IPluginHost pHost, XmlNodepInstanceData, ILogWriter pLog); 
Initialize a plugin instance by giving it the environment and any instance data specified where the plugin was referenced. Plugins should have a public default constructor so the host can activate them and do the majority of initialzation in this method. 
 
ProcessResult enum 
Result codes that may be returned from a processor 
Values 
Success 
The operation succeeded 
Failed 
The operation failed and should not be tried again 
TransientError 
The operation did not succeed but may be tried again 
 
ILogWriter interface 
Interface for log object. 
Methods 
void Log(string pMessage, LogLevel pSeverity); 
Write a message to the DIM log. 
 
LogLevel enum 
Severity levels for log messages 
 
Values 
Trace 
Messages that aid development or provide verbose processing information 
Info 
Messages that are of interest during normal processing 
Error 
Messages that indicate failure or unexpected conditions during processing 
 
IDispatchProcessor interface 
implements IDisposable 
Processes a dispatch 
Methods 
ProcessResult Execute(IDispatch pData,IVariableBindings pEnvironment, ILogWriter pLog); 
Process a dispatch 
Returns: Success if processing should continue; Failed if processing failed and should not be retried; TransientError if processing failed but may be retried later. 
ProcessResult Commit(ILogWriter pLog); 
Finalize results generated during the execute phase. 
Implementors may choose to commit results immediately in Execute rather than using a separate commit phase. 
Returns: Success if processing should continue; Failed if processing failed and should not be retried; TransientError if processing failed but may be retried later. 
void Rollback(ILogWriter pLog); 
Undo unfinalized results generated during the execute phase. 
Rollback can be called after Commit has already been called if a later processor fails to commit. In this case it's not expected that committed results will be undone, though implementors may undo them anyway if possible. 
Rollback methods should avoid throwing exceptions since this will exit the rollback phase early. If that happens then the processors that have not been called yet will never have their rollback methods invoked. 



Sample Plugin Files 

The following files are sample and definition files for use with the DIM. They can be downloaded in a single zip file from this location:  

Here is a breakdown of the files that are included in the zip file which are then broken down further in the sections below: 
  1. LoggingPlugin.cs 
  2. SqlCeWriterPlugin.cs 
  3. Sample DIM Project File 
  4. Field2Base Sample Form.xml 

Sample File: LoggingPlugin.cs 

Here is a sample DIM Plugin that writes what gets processed by the DIM to a log file. 
  1. using System; 
  2. using System.Xml; 
  3. using Field2Base.Integration.Sdk; 
  4. using Field2Base.Transformer.Sdk; 
  5.  
  6. namespace Field2Base.Transformer.Samples 
  7.     /// <summary> 
  8.     /// LoggingPlugin is a minimal plugin implementation that writes to the DIM log 
  9.     /// as a dispatch passes through each stage of processing. 
  10.     /// </summary> 
  11.     public class LoggingPlugin : IPlugin2, IDispatchProcessor 
  12.     { 
  13.         /// <summary> 
  14.         /// Plugins require a public parameterless constructor. The DIM will call this 
  15.         /// constructor to create a new plugin instance each time it is referenced by 
  16.         /// a DIM project. 
  17.         /// </summary> 
  18.         public LoggingPlugin() 
  19.         { 
  20.         } 
  21.  
  22.         /// <summary> 
  23.         /// The IDispatchProcessor interface inherits IDisposable. Implement a Dispose 
  24.         /// method to release resources as necessary and do any cleanup at shutdown. 
  25.         /// </summary> 
  26.         public void Dispose() 
  27.         { 
  28.         } 
  29.  
  30.         /// <summary> 
  31.         /// Perform one-time initialization at startup and load configuration from the 
  32.         /// project XML file. 
  33.         /// </summary> 
  34.         public void Initialize(IPluginHost pHost, XmlNode pNode, ILogWriter pLog) 
  35.         { 
  36.             pLog.Log("LoggingPlugin initialized.", LogLevel.Info); 
  37.         } 
  38.  
  39.         /// <summary> 
  40.         /// Process dispatch data. 
  41.         /// </summary> 
  42.         public ProcessResult Execute(IDispatch pData, IVariableBindings pEnvironment, ILogWriter pLog) 
  43.         { 
  44.             pLog.Log(String.Format("LoggingPlugin received dispatch {0} with ref# {1}", pEnvironment.Lookup("internalFormId"), pData.ReferenceNumber), LogLevel.Info); 
  45.             return ProcessResult.Success; 
  46.         } 
  47.  
  48.         /// <summary> 
  49.         /// If the plugin wants to participate in the transaction that coordinates multiple 
  50.         /// processing steps, it should defer finalization of any steps taken in Execute 
  51.         /// until this method gets called. 
  52.         ///  
  53.         /// Plugins are not requried to work in the transaction and may simply finalize all 
  54.         /// work in the Execute phase. 
  55.         /// </summary> 
  56.         public ProcessResult Commit(ILogWriter pLog) 
  57.         { 
  58.             pLog.Log("LoggingPlugin commit", LogLevel.Info); 
  59.             return ProcessResult.Success; 
  60.         } 
  61.  
  62.         /// <summary> 
  63.         /// If the plugin works transactionally and some later processing step encounters 
  64.         /// an error, the rollback phase allows it to discard any changes prepared during 
  65.         /// the execute phase. 
  66.         ///  
  67.         /// If the error occurs during commit of a later processing step, the plugin's 
  68.         /// Rollback method may be called after the Commit method has already been called. 
  69.         /// </summary> 
  70.         public void Rollback(ILogWriter pLog) 
  71.         { 
  72.             pLog.Log("LoggingPlugin rollback", LogLevel.Info); 
  73.         } 
  74.     } 

 

Sample File: SqlCeWriterPlugin.cs 

Here is a sample plugin that connects to a SQL Compact Edition database and writes region values to the database. NOTE: This plugin requires that Microsoft SQL Compact Edition 3.5 be installed where the DIM is running. 
  1. using System; 
  2. using System.IO; 
  3. using System.Xml; 
  4. using System.Data.SqlServerCe; 
  5. using Field2Base.Integration.Sdk; 
  6. using Field2Base.Transformer.Sdk; 
  7.  
  8. namespace Field2Base.Transformer.Samples 
  9.     /// <summary> 
  10.     /// SqlCeWriterPlugin is a plugin implementation that writes form data to a SQL CE database. 
  11.     /// </summary> 
  12.     public class SqlCeWriterPlugin : IPlugin2, IDispatchProcessor 
  13.     { 
  14.         public SqlCeWriterPlugin() 
  15.         { 
  16.         } 
  17.  
  18.         private string _dbFilePath;   // path to SQL CE database that will hold form data 
  19.         private SqlCeConnection _conn;   // db connection while processing a form 
  20.         private SqlCeTransaction _tran;   // db transaction while processing a form 
  21.  
  22.         private string DbConnectionString 
  23.         { 
  24.             get { return String.Format("Data Source = {0}", _dbFilePath); } 
  25.         } 
  26.  
  27.         public void Dispose() 
  28.         { 
  29.             DisposeDbObjects(); 
  30.         } 
  31.  
  32.         /// <summary> 
  33.         /// Initialize this plugin instance. 
  34.         /// </summary> 
  35.         public void Initialize(IPluginHost pHost, XmlNode pInstanceData, ILogWriter pLogWriter) 
  36.         { 
  37.             // make sure we have required configuration 
  38.             if (pInstanceData["dbFilePath"] == null) 
  39.                 throw new ApplicationException("SqlCeWriterPlugin instance data is missing the 'dbFilePath' element."); 
  40.  
  41.             // load configuration 
  42.             _dbFilePath = pInstanceData["dbFilePath"].InnerText; 
  43.  
  44.             // create the db that will receive form data if it doesn't already exist 
  45.             CreateDatabaseIfNecessary(); 
  46.         } 
  47.  
  48.         /// <summary> 
  49.         /// Process dispatch data. 
  50.         /// </summary> 
  51.         public ProcessResult Execute(IDispatch pDispatchData, IVariableBindings pVariables, ILogWriter pLog) 
  52.         { 
  53.             // open the db, create a transaction, and populate for this form 
  54.             try 
  55.             { 
  56.                 _conn = new SqlCeConnection(DbConnectionString); 
  57.                 _conn.Open(); 
  58.                 _tran = _conn.BeginTransaction(); 
  59.  
  60.                 foreach (ISentFormData formData in pDispatchData.Forms) 
  61.                 { 
  62.                     // the DIM project can be configured to skip form data, so make sure we have a data document 
  63.                     if (formData.Data == null) continue; 
  64.  
  65.                     // get id assigned by the client when the form was sent 
  66.                     string formDataIdStr = formData.GetHeaderValue("FormDataId"); 
  67.  
  68.                     // insert a row for the form 
  69.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  70.                     { 
  71.                         cmd.Connection = _conn; 
  72.                         cmd.Transaction = _tran; 
  73.                         cmd.CommandText = "INSERT INTO Form ( FormDataId ) VALUES ( @formDataId )"; 
  74.                         cmd.Parameters.AddWithValue("@formDataId", formDataIdStr); 
  75.                         cmd.ExecuteNonQuery(); 
  76.                     } 
  77.  
  78.                     // get the identity value for the form 
  79.                     int formId; 
  80.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  81.                     { 
  82.                         cmd.Connection = _conn; 
  83.                         cmd.Transaction = _tran; 
  84.                         cmd.CommandText = "SELECT @@IDENTITY"; 
  85.                         formId = Convert.ToInt32(cmd.ExecuteScalar()); 
  86.                     } 
  87.                      
  88.                     // insert region values 
  89.                     foreach (XmlNode regionNode in formData.Data.SelectNodes("/FormData/Values/Page/Region")) 
  90.                     { 
  91.                         int userPageNumber = Int32.Parse(regionNode.ParentNode.Attributes["UserPageNumber"].Value); 
  92.                         string regionName = regionNode.Attributes["Name"].Value; 
  93.                         string regionFullName = String.Format("Page{0}@{1}", userPageNumber, regionName); 
  94.  
  95.                         string regionValue = formData.GetRegionValue(regionFullName); 
  96.  
  97.                         using (SqlCeCommand cmd = new SqlCeCommand()) 
  98.                         { 
  99.                             cmd.Connection = _conn; 
  100.                             cmd.Transaction = _tran; 
  101.                             cmd.CommandText = "INSERT INTO FormData ( ID, UserPageNumber, RegionName, RegionValue ) VALUES ( @id, @userPageNumber, @regionName, @regionValue )"; 
  102.                             cmd.Parameters.AddWithValue("@id", formId); 
  103.                             cmd.Parameters.AddWithValue("@userPageNumber", userPageNumber); 
  104.                             cmd.Parameters.AddWithValue("@regionName", regionName); 
  105.                             cmd.Parameters.AddWithValue("@regionValue", regionValue); 
  106.                             cmd.ExecuteNonQuery(); 
  107.                         } 
  108.                     } 
  109.                 } 
  110.             } 
  111.             catch 
  112.             { 
  113.                 // must clean up db objects in case of error because we won't get a later Rollback or Commit 
  114.                 //   if we don't return success from this method 
  115.                 DisposeDbObjects(); 
  116.                 throw; 
  117.             } 
  118.  
  119.             return ProcessResult.Success; 
  120.         } 
  121.  
  122.         /// <summary> 
  123.         /// Commit the transaction once all processing steps are complete. 
  124.         /// </summary> 
  125.         public ProcessResult Commit(ILogWriter pLog) 
  126.         { 
  127.             // try to commit the transaction 
  128.             try 
  129.             { 
  130.                 _tran.Commit(); 
  131.                 pLog.Log("SQL CE transaction has been committed.", LogLevel.Trace); 
  132.                 return ProcessResult.Success; 
  133.             } 
  134.             catch (SqlCeException ex) 
  135.             { 
  136.                 pLog.Log(String.Format("Could not commit SQL CE transaction. Caught {0}: {1}", ex.GetType().Name, ex.Message), LogLevel.Error); 
  137.                 return ProcessResult.Failed; 
  138.             } 
  139.             finally 
  140.             { 
  141.                 DisposeDbObjects(); 
  142.             } 
  143.         } 
  144.  
  145.         /// <summary> 
  146.         /// Roll back the transaction if a later processing step failed. 
  147.         /// </summary> 
  148.         public void Rollback(ILogWriter pLog) 
  149.         { 
  150.             // try to roll back the transaction 
  151.             try 
  152.             { 
  153.                 // must check for null transaction because Rollback will be called after Commit 
  154.                 //   if a later processing step fails during its Commit phase 
  155.                 if (_tran != null) 
  156.                 { 
  157.                     _tran.Rollback(); 
  158.                     pLog.Log("SQL CE transaction has been rolled back.", LogLevel.Trace); 
  159.                 } 
  160.                 else 
  161.                 { 
  162.                     pLog.Log("SQL CE transaction is already complete and cannot be rolled back.", LogLevel.Info); 
  163.                 } 
  164.             } 
  165.             finally 
  166.             { 
  167.                 DisposeDbObjects(); 
  168.             } 
  169.         } 
  170.         private void DisposeDbObjects() 
  171.         { 
  172.             if (_tran != null) 
  173.             { 
  174.                 _tran.Dispose(); 
  175.                 _tran = null; 
  176.             } 
  177.             if (_conn != null) 
  178.             { 
  179.                 _conn.Dispose(); 
  180.                 _conn = null; 
  181.             } 
  182.         } 
  183.         private void CreateDatabaseIfNecessary() 
  184.         { 
  185.             // nothing to do if the file has already been created 
  186.             if (File.Exists(_dbFilePath)) 
  187.                 return; 
  188.  
  189.             // create a new db file 
  190.             using (SqlCeEngine engine = new SqlCeEngine(DbConnectionString)) 
  191.             { 
  192.                 engine.CreateDatabase(); 
  193.             } 
  194.  
  195.             // create the data tables 
  196.             try 
  197.             { 
  198.                 using (SqlCeConnection conn = new SqlCeConnection(DbConnectionString)) 
  199.                 { 
  200.                     conn.Open(); 
  201.  
  202.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  203.                     { 
  204.                         cmd.Connection = conn; 
  205.                         cmd.CommandText = @" 
  206.                         CREATE TABLE Form ( 
  207.                             ID int identity(1, 1), 
  208.                             FormDataId uniqueidentifier 
  209.                         )"; 
  210.                         cmd.ExecuteNonQuery(); 
  211.                     } 
  212.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  213.                     { 
  214.                         cmd.Connection = conn; 
  215.                         cmd.CommandText = @" 
  216.                         CREATE TABLE FormData ( 
  217.                             ID int, 
  218.                             UserPageNumber int, 
  219.                             RegionName ntext, 
  220.                             RegionValue ntext 
  221.                         )"; 
  222.                         cmd.ExecuteNonQuery(); 
  223.                     } 
  224.                 } 
  225.             } 
  226.             catch 
  227.             { 
  228.                 // don't leave a half-created db file 
  229.                 File.Delete(_dbFilePath); 
  230.                 throw; 
  231.             } 
  232.         } 
  233.     } 

 

Project File: Field2Base Sample Form.xml 

Here is a Sample DIM Project file that is using both the LoggingPlugin and SqlCeWriterPlugin plugins. 
 
  1. <?xml version="1.0" encoding="utf-8"?> 
  2. <processForms> 
  3.   <systemName>Field2Base Sample Form</systemName> 
  4.   <friendlyName>Field2Base Sample Form</friendlyName> 
  5.   <formFileName>Field2Base Sample Form.eform</formFileName> 
  6.   <enabled>True</enabled> 
  7.   <rfiSvc> 
  8.     <name>Field2Base Sample Form</name> 
  9.     <uri>https://fieldconnect.field2base.com:2443/V2/F2BRFIWebService/GetRfi.asmx%3C/uri%3E 
  10.     <username>RfiSvcClient</username> 
  11.     <password>password</password> 
  12.     <companyId>101</companyId> 
  13.     <formTemplateId>80d54d03-cec0-472a-9400-b6c8f17582ac</formTemplateId> 
  14.   </rfiSvc> 
  15.   <processPlugin class="Field2Base.Transformer.Samples.LoggingPlugin,DataIntegrationSamples" /> 
  16.   <processPlugin class="Field2Base.Transformer.Samples.SqlCeWriterPlugin,DataIntegrationSamples"> 
  17.     <dbFilePath>c:\FormData.sdf</dbFilePath> 
  18.   </processPlugin> 
  19. </processForms>

    • Related Articles

    • DIM PDF Plugin Setup

      Overview The PDF Plugin allows customers running the Data Integration Manager (DIM) to save PDF files of Sent Forms to a specified file location. There are limited customize-able options for this plugin. The following options are available: The ...
    • DIM Image Plugin Set Up Guide

      Overview This document will outline the process for setting up the Image Plugin created by Field2Base for use with the Data Integration Module (DIM). The Image Plugin allows a Customer running the DIM to generate Image files (JPG, PNG, etc.) from ...
    • Data Integration Module (DIM) User Guide

      Overview The Field2Base Data Integration Module allows you to automatically download data from your Sent Forms to your backend systems. The DIM provides transformation of your Form Data into a variety of formats including CSV and SQL. The DIM Sample ...
    • DIM Reset Guide

      Overview This document will familiarize you with the process behind resetting your installation of the Field2Base Data Integration Module (DIM). You may want to reset the DIM if you wish to recapture all data starting from a specific point in time ...
    • DIM Migration Guide

      Overview This article provides the information necessary to migrate the Field2Base Data Integration Module (DIM) over from one server to another.  Please refer to the DIM Install Guide for the initial installation of the Field2Base DIM.  Once that's ...