Data Integration Module (DIM) Custom Plugin Reference Guide

Data Integration Module (DIM) Custom Plugin Reference Guide

Overview

The Data Integration Module (DIM) allows Users to create custom Plugins that can directly interface with the Data Integration Service and Projects. This document describes how to setup and use Plugins with the DIM. 

DIM Processing Overview 

The Data Integration Module executes processing instructions from top to bottom as defined in the XML project file. First, it sends each processor an ‘execute’ message in order. After each processor indicates successful execution, it sends a ‘commit’ message to finalize changes. This sequence establishes a transaction for atomic dispatch processing. However, processors are not required to participate in the transaction and may finalize changes directly during the execute phase. 

If any processor fails to execute or commit successfully, the DIM sends a ‘rollback’ message to all processors that completed execution. Because an error may happen during the commit phase, it is possible for a processor to receive a rollback message after it has already received a commit message. Most processors will simply ignore a rollback message received after a commit message.
 

Plugin Details and Requirements 

You can include custom data processing code in DIM execution by writing a .NET 2.0 class implementing the IPlugin2 and IDispatchProcessor interfaces from TransformerSdk.dll. Add your plugin to the processing sequence using a processPlugin element in the project XML file. Your plugin assembly and any other libs it depends on should be copied into the DIM Program Files folder. The default location for the DIM is one of following: 
  1. 32 Bit OS: C:\Program Files\F2B Data Integration Module 
  2. 64 Bit OS: C:\Program Files (x86)\F2B Data Integration Module 
The processPlugin element must give the assembly-qualified name of your plugin class using a class attribute. 

IPlugin2 defines an Initialize method that your plugin should implement to do any startup tasks when the DIM loads the project file. The plugin receives the processPlugin XML element from the project XML file, which can be used for instance configuration. For example, a plugin that creates files may specify the location for new files as a child of processPlugin. The plugin also receives an IPluginHost containing a WaitHandle that gets signaled when the DIM service is shutting down. 

IDispatchProcessor defines methods Execute, Commit, and Rollback. Implement these methods to execute your custom processing code at each stage of processing. 
The Execute method receives dispatch data using the IDispatch interface. This interface contains some information such as the dispatch reference number and sender’s username. It also lists the rendered files included in the dispatch and contains a list of ISentFormData instances providing access to sent form data. Most dispatches will simply contain one sent form. 

DIM Interface Definitions 

The following section outlines the definitions for the Integration and Transformer SDK’s of the DIM Plugin API. 

Field2Base.Integration.Sdk Namespace 
IDispatch Interface 
Accesses dispatch data 
Properties 
int ReferenceNumber { get; } 
Get the reference number assigned to this dispatch by F2B 
string SenderUsername { get; } 
Get the username of the user who sent this dispatch 
IList<ISentFormData> Forms { get; } 
Get form data included in this dispatch 
IList<FileInfo> ContentFiles { get; } 
Get a list of files included in the dispatch 
 
ISentFormData Interface 
Accesses form data 
 
Properties 
XmlDocument Data { get; } 
Direct access to the XML data file. Null if the data file is missing. 
 
Methods 
string GetRegionValue(string pRegionFullName); 
Helper to get a region value 
Returns: Serialized string region value; null if the region doesn't exist or was left blank 
string GetHeaderValue(string pHeaderKey); 
Helper to get a header value 
Returns: Header value; null if the value doesn't exist 
byte[] GetPageImage(int pPageIndex); 
Get image data for a rendered form page using its 0-based index in the form 
Returns: Page image; null if the page doesn't exist or page image data isn't available 
 
IVariableBindings Interface 
Carries execution parameters and variables during form processing. i.e. The min and max date values for restricting processing to a date range. 
Methods 
object Lookup(string pIdentifier); 
Get the value bound to an identifier 
Returns: The value bound to an identifier; null if the identifier is not bound. 
 
Field2Base.Transformer.Sdk Namespace 
IPluginHost interface 
Provides an environment for plugins and form processing in general 
Properties 
WaitHandle StopSignal { get; } 
Signal set when the host is trying to shut down 
 
IPlugin2 interface 
Generic interface for extensions 
Methods 
void Initialize(IPluginHost pHost, XmlNodepInstanceData, ILogWriter pLog); 
Initialize a plugin instance by giving it the environment and any instance data specified where the plugin was referenced. Plugins should have a public default constructor so the host can activate them and do the majority of initialzation in this method. 
 
ProcessResult enum 
Result codes that may be returned from a processor 
Values 
Success 
The operation succeeded 
Failed 
The operation failed and should not be tried again 
TransientError 
The operation did not succeed but may be tried again 
 
ILogWriter interface 
Interface for log object. 
Methods 
void Log(string pMessage, LogLevel pSeverity); 
Write a message to the DIM log. 
 
LogLevel enum 
Severity levels for log messages 
 
Values 
Trace 
Messages that aid development or provide verbose processing information 
Info 
Messages that are of interest during normal processing 
Error 
Messages that indicate failure or unexpected conditions during processing 
 
IDispatchProcessor interface 
implements IDisposable 
Processes a dispatch 
Methods 
ProcessResult Execute(IDispatch pData,IVariableBindings pEnvironment, ILogWriter pLog); 
Process a dispatch 
Returns: Success if processing should continue; Failed if processing failed and should not be retried; TransientError if processing failed but may be retried later. 
ProcessResult Commit(ILogWriter pLog); 
Finalize results generated during the execute phase. 
Implementors may choose to commit results immediately in Execute rather than using a separate commit phase. 
Returns: Success if processing should continue; Failed if processing failed and should not be retried; TransientError if processing failed but may be retried later. 
void Rollback(ILogWriter pLog); 
Undo unfinalized results generated during the execute phase. 
Rollback can be called after Commit has already been called if a later processor fails to commit. In this case it's not expected that committed results will be undone, though implementors may undo them anyway if possible. 
Rollback methods should avoid throwing exceptions since this will exit the rollback phase early. If that happens then the processors that have not been called yet will never have their rollback methods invoked. 


Sample Plugin Files 

The following files are sample and definition files for use with the DIM. They can be downloaded in a single zip file from this location:  

Here is a breakdown of the files that are included in the zip file which are then broken down further in the sections below: 
  1. LoggingPlugin.cs 
  2. SqlCeWriterPlugin.cs 
  3. Sample DIM Project File 
  4. Field2Base Sample Form.xml 

Sample File: LoggingPlugin.cs 

Here is a sample DIM Plugin that writes what gets processed by the DIM to a log file. 
  1. using System; 
  2. using System.Xml; 
  3. using Field2Base.Integration.Sdk; 
  4. using Field2Base.Transformer.Sdk; 
  5.  
  6. namespace Field2Base.Transformer.Samples 
  7. { 
  8.     /// <summary> 
  9.     /// LoggingPlugin is a minimal plugin implementation that writes to the DIM log 
  10.     /// as a dispatch passes through each stage of processing. 
  11.     /// </summary> 
  12.     public class LoggingPlugin : IPlugin2, IDispatchProcessor 
  13.     { 
  14.         /// <summary> 
  15.         /// Plugins require a public parameterless constructor. The DIM will call this 
  16.         /// constructor to create a new plugin instance each time it is referenced by 
  17.         /// a DIM project. 
  18.         /// </summary> 
  19.         public LoggingPlugin() 
  20.         { 
  21.         } 
  22.  
  23.         /// <summary> 
  24.         /// The IDispatchProcessor interface inherits IDisposable. Implement a Dispose 
  25.         /// method to release resources as necessary and do any cleanup at shutdown. 
  26.         /// </summary> 
  27.         public void Dispose() 
  28.         { 
  29.         } 
  30.  
  31.         /// <summary> 
  32.         /// Perform one-time initialization at startup and load configuration from the 
  33.         /// project XML file. 
  34.         /// </summary> 
  35.         public void Initialize(IPluginHost pHost, XmlNode pNode, ILogWriter pLog) 
  36.         { 
  37.             pLog.Log("LoggingPlugin initialized.", LogLevel.Info); 
  38.         } 
  39.  
  40.         /// <summary> 
  41.         /// Process dispatch data. 
  42.         /// </summary> 
  43.         public ProcessResult Execute(IDispatch pData, IVariableBindings pEnvironment, ILogWriter pLog) 
  44.         { 
  45.             pLog.Log(String.Format("LoggingPlugin received dispatch {0} with ref# {1}", pEnvironment.Lookup("internalFormId"), pData.ReferenceNumber), LogLevel.Info); 
  46.             return ProcessResult.Success; 
  47.         } 
  48.  
  49.         /// <summary> 
  50.         /// If the plugin wants to participate in the transaction that coordinates multiple 
  51.         /// processing steps, it should defer finalization of any steps taken in Execute 
  52.         /// until this method gets called. 
  53.         ///  
  54.         /// Plugins are not requried to work in the transaction and may simply finalize all 
  55.         /// work in the Execute phase. 
  56.         /// </summary> 
  57.         public ProcessResult Commit(ILogWriter pLog) 
  58.         { 
  59.             pLog.Log("LoggingPlugin commit", LogLevel.Info); 
  60.             return ProcessResult.Success; 
  61.         } 
  62.  
  63.         /// <summary> 
  64.         /// If the plugin works transactionally and some later processing step encounters 
  65.         /// an error, the rollback phase allows it to discard any changes prepared during 
  66.         /// the execute phase. 
  67.         ///  
  68.         /// If the error occurs during commit of a later processing step, the plugin's 
  69.         /// Rollback method may be called after the Commit method has already been called. 
  70.         /// </summary> 
  71.         public void Rollback(ILogWriter pLog) 
  72.         { 
  73.             pLog.Log("LoggingPlugin rollback", LogLevel.Info); 
  74.         } 
  75.     } 
  76. } 
 

Sample File: SqlCeWriterPlugin.cs 

Here is a sample plugin that connects to a SQL Compact Edition database and writes region values to the database. NOTE: This plugin requires that Microsoft SQL Compact Edition 3.5 be installed where the DIM is running. 
  1. using System; 
  2. using System.IO; 
  3. using System.Xml; 
  4. using System.Data.SqlServerCe; 
  5. using Field2Base.Integration.Sdk; 
  6. using Field2Base.Transformer.Sdk; 
  7.  
  8. namespace Field2Base.Transformer.Samples 
  9. { 
  10.     /// <summary> 
  11.     /// SqlCeWriterPlugin is a plugin implementation that writes form data to a SQL CE database. 
  12.     /// </summary> 
  13.     public class SqlCeWriterPlugin : IPlugin2, IDispatchProcessor 
  14.     { 
  15.         public SqlCeWriterPlugin() 
  16.         { 
  17.         } 
  18.  
  19.         private string _dbFilePath;   // path to SQL CE database that will hold form data 
  20.         private SqlCeConnection _conn;   // db connection while processing a form 
  21.         private SqlCeTransaction _tran;   // db transaction while processing a form 
  22.  
  23.         private string DbConnectionString 
  24.         { 
  25.             get { return String.Format("Data Source = {0}", _dbFilePath); } 
  26.         } 
  27.  
  28.         public void Dispose() 
  29.         { 
  30.             DisposeDbObjects(); 
  31.         } 
  32.  
  33.         /// <summary> 
  34.         /// Initialize this plugin instance. 
  35.         /// </summary> 
  36.         public void Initialize(IPluginHost pHost, XmlNode pInstanceData, ILogWriter pLogWriter) 
  37.         { 
  38.             // make sure we have required configuration 
  39.             if (pInstanceData["dbFilePath"] == null) 
  40.                 throw new ApplicationException("SqlCeWriterPlugin instance data is missing the 'dbFilePath' element."); 
  41.  
  42.             // load configuration 
  43.             _dbFilePath = pInstanceData["dbFilePath"].InnerText; 
  44.  
  45.             // create the db that will receive form data if it doesn't already exist 
  46.             CreateDatabaseIfNecessary(); 
  47.         } 
  48.  
  49.         /// <summary> 
  50.         /// Process dispatch data. 
  51.         /// </summary> 
  52.         public ProcessResult Execute(IDispatch pDispatchData, IVariableBindings pVariables, ILogWriter pLog) 
  53.         { 
  54.             // open the db, create a transaction, and populate for this form 
  55.             try 
  56.             { 
  57.                 _conn = new SqlCeConnection(DbConnectionString); 
  58.                 _conn.Open(); 
  59.                 _tran = _conn.BeginTransaction(); 
  60.  
  61.                 foreach (ISentFormData formData in pDispatchData.Forms) 
  62.                 { 
  63.                     // the DIM project can be configured to skip form data, so make sure we have a data document 
  64.                     if (formData.Data == null) continue; 
  65.  
  66.                     // get id assigned by the client when the form was sent 
  67.                     string formDataIdStr = formData.GetHeaderValue("FormDataId"); 
  68.  
  69.                     // insert a row for the form 
  70.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  71.                     { 
  72.                         cmd.Connection = _conn; 
  73.                         cmd.Transaction = _tran; 
  74.                         cmd.CommandText = "INSERT INTO Form ( FormDataId ) VALUES ( @formDataId )"; 
  75.                         cmd.Parameters.AddWithValue("@formDataId", formDataIdStr); 
  76.                         cmd.ExecuteNonQuery(); 
  77.                     } 
  78.  
  79.                     // get the identity value for the form 
  80.                     int formId; 
  81.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  82.                     { 
  83.                         cmd.Connection = _conn; 
  84.                         cmd.Transaction = _tran; 
  85.                         cmd.CommandText = "SELECT @@IDENTITY"; 
  86.                         formId = Convert.ToInt32(cmd.ExecuteScalar()); 
  87.                     } 
  88.                      
  89.                     // insert region values 
  90.                     foreach (XmlNode regionNode in formData.Data.SelectNodes("/FormData/Values/Page/Region")) 
  91.                     { 
  92.                         int userPageNumber = Int32.Parse(regionNode.ParentNode.Attributes["UserPageNumber"].Value); 
  93.                         string regionName = regionNode.Attributes["Name"].Value; 
  94.                         string regionFullName = String.Format("Page{0}@{1}", userPageNumber, regionName); 
  95.  
  96.                         string regionValue = formData.GetRegionValue(regionFullName); 
  97.  
  98.                         using (SqlCeCommand cmd = new SqlCeCommand()) 
  99.                         { 
  100.                             cmd.Connection = _conn; 
  101.                             cmd.Transaction = _tran; 
  102.                             cmd.CommandText = "INSERT INTO FormData ( ID, UserPageNumber, RegionName, RegionValue ) VALUES ( @id, @userPageNumber, @regionName, @regionValue )"; 
  103.                             cmd.Parameters.AddWithValue("@id", formId); 
  104.                             cmd.Parameters.AddWithValue("@userPageNumber", userPageNumber); 
  105.                             cmd.Parameters.AddWithValue("@regionName", regionName); 
  106.                             cmd.Parameters.AddWithValue("@regionValue", regionValue); 
  107.                             cmd.ExecuteNonQuery(); 
  108.                         } 
  109.                     } 
  110.                 } 
  111.             } 
  112.             catch 
  113.             { 
  114.                 // must clean up db objects in case of error because we won't get a later Rollback or Commit 
  115.                 //   if we don't return success from this method 
  116.                 DisposeDbObjects(); 
  117.                 throw; 
  118.             } 
  119.  
  120.             return ProcessResult.Success; 
  121.         } 
  122.  
  123.         /// <summary> 
  124.         /// Commit the transaction once all processing steps are complete. 
  125.         /// </summary> 
  126.         public ProcessResult Commit(ILogWriter pLog) 
  127.         { 
  128.             // try to commit the transaction 
  129.             try 
  130.             { 
  131.                 _tran.Commit(); 
  132.                 pLog.Log("SQL CE transaction has been committed.", LogLevel.Trace); 
  133.                 return ProcessResult.Success; 
  134.             } 
  135.             catch (SqlCeException ex) 
  136.             { 
  137.                 pLog.Log(String.Format("Could not commit SQL CE transaction. Caught {0}: {1}", ex.GetType().Name, ex.Message), LogLevel.Error); 
  138.                 return ProcessResult.Failed; 
  139.             } 
  140.             finally 
  141.             { 
  142.                 DisposeDbObjects(); 
  143.             } 
  144.         } 
  145.  
  146.         /// <summary> 
  147.         /// Roll back the transaction if a later processing step failed. 
  148.         /// </summary> 
  149.         public void Rollback(ILogWriter pLog) 
  150.         { 
  151.             // try to roll back the transaction 
  152.             try 
  153.             { 
  154.                 // must check for null transaction because Rollback will be called after Commit 
  155.                 //   if a later processing step fails during its Commit phase 
  156.                 if (_tran != null) 
  157.                 { 
  158.                     _tran.Rollback(); 
  159.                     pLog.Log("SQL CE transaction has been rolled back.", LogLevel.Trace); 
  160.                 } 
  161.                 else 
  162.                 { 
  163.                     pLog.Log("SQL CE transaction is already complete and cannot be rolled back.", LogLevel.Info); 
  164.                 } 
  165.             } 
  166.             finally 
  167.             { 
  168.                 DisposeDbObjects(); 
  169.             } 
  170.         } 
  171.         private void DisposeDbObjects() 
  172.         { 
  173.             if (_tran != null) 
  174.             { 
  175.                 _tran.Dispose(); 
  176.                 _tran = null; 
  177.             } 
  178.             if (_conn != null) 
  179.             { 
  180.                 _conn.Dispose(); 
  181.                 _conn = null; 
  182.             } 
  183.         } 
  184.         private void CreateDatabaseIfNecessary() 
  185.         { 
  186.             // nothing to do if the file has already been created 
  187.             if (File.Exists(_dbFilePath)) 
  188.                 return; 
  189.  
  190.             // create a new db file 
  191.             using (SqlCeEngine engine = new SqlCeEngine(DbConnectionString)) 
  192.             { 
  193.                 engine.CreateDatabase(); 
  194.             } 
  195.  
  196.             // create the data tables 
  197.             try 
  198.             { 
  199.                 using (SqlCeConnection conn = new SqlCeConnection(DbConnectionString)) 
  200.                 { 
  201.                     conn.Open(); 
  202.  
  203.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  204.                     { 
  205.                         cmd.Connection = conn; 
  206.                         cmd.CommandText = @" 
  207.                         CREATE TABLE Form ( 
  208.                             ID int identity(1, 1), 
  209.                             FormDataId uniqueidentifier 
  210.                         )"; 
  211.                         cmd.ExecuteNonQuery(); 
  212.                     } 
  213.                     using (SqlCeCommand cmd = new SqlCeCommand()) 
  214.                     { 
  215.                         cmd.Connection = conn; 
  216.                         cmd.CommandText = @" 
  217.                         CREATE TABLE FormData ( 
  218.                             ID int, 
  219.                             UserPageNumber int, 
  220.                             RegionName ntext, 
  221.                             RegionValue ntext 
  222.                         )"; 
  223.                         cmd.ExecuteNonQuery(); 
  224.                     } 
  225.                 } 
  226.             } 
  227.             catch 
  228.             { 
  229.                 // don't leave a half-created db file 
  230.                 File.Delete(_dbFilePath); 
  231.                 throw; 
  232.             } 
  233.         } 
  234.     } 
  235. } 
 

Project File: Field2Base Sample Form.xml 

Here is a Sample DIM Project file that is using both the LoggingPlugin and SqlCeWriterPlugin plugins. 
 
  1. <?xml version="1.0" encoding="utf-8"?> 
  2. <processForms> 
  3.   <systemName>Field2Base Sample Form</systemName> 
  4.   <friendlyName>Field2Base Sample Form</friendlyName> 
  5.   <formFileName>Field2Base Sample Form.eform</formFileName> 
  6.   <enabled>True</enabled> 
  7.   <rfiSvc> 
  8.     <name>Field2Base Sample Form</name> 
  9.     <uri>https://fieldconnect.field2base.com/V2/F2BRFIWebService/GetRfi.asmx</uri>
  10.     <username>RfiSvcClient</username> 
  11.     <password>password</password> 
  12.     <companyId>101</companyId> 
  13.     <formTemplateId>80d54d03-cec0-472a-9400-b6c8f17582ac</formTemplateId> 
  14.   </rfiSvc> 
  15.   <processPlugin class="Field2Base.Transformer.Samples.LoggingPlugin,DataIntegrationSamples" /> 
  16.   <processPlugin class="Field2Base.Transformer.Samples.SqlCeWriterPlugin,DataIntegrationSamples"> 
  17.     <dbFilePath>c:\FormData.sdf</dbFilePath> 
  18.   </processPlugin> 
  19. </processForms>

    • Popular Articles

    • Forms Designer Quick Start Guide

      Overview Field2Base Forms Designer is the proprietary software application that allows your existing paper forms to be quickly converted to a smart E-form available to your end users via our mobile and web-based Mobile Forms applications. This ...

    • Portal 11.28.2023 Release Notes

      Overview Our release notes offer brief descriptions of product enhancements and bug fixes. We include links to the current articles for any affected features. Those articles will be updated shortly after the Portal release to include new ...

    • Integration Service Configuration Guide

      How To Configure Integration Services To Allow Read/Write Access on a Network Path All of our Integration Products, including the DIM, DUU, and EDM have respective Windows Services responsible for communicating with our server. Occassionally, ...

    • Data Integration Module (DIM) Migration Guide

      Overview This article provides the information necessary to migrate the Field2Base Data Integration Module (DIM) over from one server to another. Please refer to the DIM Install Guide for the initial installation of the Field2Base DIM. Once that's ...

    • How to Check the Version of Integration Products Running on a Windows 10 Machine

      Right-click on the Start menu button. Click on Apps & Features. In the Apps & Features search bar type in the Integration Product you are looking for, eg. F2B Data Integration Module, F2B Data Upload, or F2B Enterprise Dispatch Module. Click to ...