Transformer - Build a data loading program

In this section, we'll cover how to build programs that transform your raw data into structured XML files called events and send those events to your Network.

Programming Approaches

The primary approach to building data loading programs involves using the Doodlebase Transformer. Transformer is a DLL (Dynamic Link Library) containing helper methods for sending structured data to your Network.

The following programming environments work well with Transformer:

• LabVIEW by National Instruments.
• The .NET framework by Microsoft.

Prerequisites

The following examples utilize the Transformer library and are written in C#. If you'd like to follow along with your own program, send us an email at steve@schemaport.com and request an evaluation of the Transformer DLL.

Install Visual Studio

You should have the latest version of Microsoft Visual Studio installed on your computer. You can download a free version of Visual Studio called Visual Studio Community by clicking the link here.

Create a Project

Open Visual Studio, and select File > New Project > Console Application. Now, you need to select a framework. For this example, select .NET Framework.

Add Transformer to your project

Before you can start using Transformer, you need to add Transformer as a project dependency. To do this, go to your Solution Explorer and right click References, then select Add Reference.

Now, navigate to the Doodlebase Transformer DLL in your file directory and select add.

Write the program

Your data loading program should be responsible for the following activities:

• Parsing Raw Data generated by your machines/manufacturing process.
• Transforming Raw Data into structured XML files called events.
• Adding Files such as images, PDF documents, audio clips, etc.
• Loading data to your Network.
• Handling Network connectivity issues.
• Verifying data was properly loaded to your Network.

Parsing Raw Data

The first step is to read your raw data into memory. Raw data is typically generated by clients in your manufacturing process, but may also be human generated as well. Your data will likely be stored in various file types such as TSV, CSV, or XML.

Since there are several methods to parsing data, we won't cover this here. We'll assume you've found a way to access your data, and are ready to begin constructing events.

Constructing Events

If you are using Transformer, you build events by calling methods in the following order:

Create an event

We'll start by creating an event object. An event object represents the XML file your program is building.

//Create and configure event
var config = new Configuration("routeKey", "password");
var myEvent = new Event(config, DateTime.UtcNow);

An event requires a Configuration Object to be initialized. A Configuration object associates your event to a specific process step by using a process step's routeKey and password. These credentials and where to find them are detailed on the Load > Concepts page.

An event also requires a TimeStamp, representing the specific moment in time the event occurred. Ensure you are initializing your events in UTC time. Data should be stored in your database as UTC time and can be converted to local time within your BI tool.

Create a product

Now that we've set up an event, it's time to create a Product object. At a minimum, a product must have a serial number and a status describing the outcome of the event. To start, we'll initialize our product's status to PASS, but we may change it's status later based on the outcome of the event.

//Create a new product object
var product = new Product("serial_number", DbLoadProductStatus.PASS);

There are four status types you can assign to a product:

Typically, your test equipment should report a status of either PASS, FAIL, or ERROR. Use PASS and FAIL to specify the outcome of the product test. Additionally, you can use ERROR to describe an incomplete event, or an unexpected issue with the test equipment.

For events that shouldn't be associated with yield such as Assembly or Ship events, use LOG. This way, you can ignore these events when calculating yield.

Add elements to the product

Elements are the unique information describing the event itself. You'll add different elements to each event, depending on the data you wish to capture. Here are some examples of elements you can add to your product.

//Add a variable to the product
var myVariable = new Variable("name", "value", DbLoadVariableStatus.PASS);
myVariable.Usl = "value";
myVariable.Lsl = "value";

product.Add(myVariable);

//Add an attribute to the product
product.Add(new Attribute("name", "value", DbLoadAttributeStatus.PASS));

//Add a component to the product
var myComponent = new Component("part_number");
myComponent.LotCode = "lot_code";
myComponent.Manufacturer = "manufacturer";
product.Add(myComponent);

//Add a symptom to the product
product.Add(new Symptom("name", "value"));

Resolve namespace conflict

If you try to create an Attribute object, you may notice that Visual Studio notifies you of a namespace conflict with .NET's System namespace. To resolve this issue, add the following code at the top of your program:

using Attribute = Schemaport.Doodlebase.Transformer.Attribute;

Update a product’s status

Sometimes, when parsing your data, you will uncover measurements that indicate a failure. In these situations, you may want to update a product's status. You can accomplish this by reassigning a product's status property.

product.Status = DbLoadProductStatus.FAIL

Add product to event

Once you are done adding elements to a product and have finished updating the status, you are ready to add the product to the event object.

//Add product to event
myEvent.Add(product);

Optional: Add Files

You can optionally include files such as images, PDF documents, auto clips, or any binary file with your event.

You add files with Transformer by first creating a package. You can think of the package as the zip folder containing the files you wish to include with your event. Once your package is created, you can call the add method to include files in your package.

//Create a ZIP folder and include the event XML
var pkg = new Package(myEvent);

//Add a file to the ZIP folder
string imageFilePath = @"C:\images\image1.jpg";
pkg.AddFile(imageFilePath);

Keep in mind that the maximum size of a package is 5MB. Packages that exceed this limit will fail to load data to your Network.

Load your Data

The easiest way to load your event or package is to simply call the load method;

//load the event
myEvent.load();

//or load a package
pkg.load();

Note that package objects have the same load methods as events. For brevity, event objects will be used in the loading examples below.

Every load event returns a loadinfo object detailing the result of the load. It's a common practice to capture this object to ensure your loads are successful.

//Capture information about a load
var loadInfo = myEvent.load()

The loadinfo object has several useful properties that allow you to perform tasks such as:

• Counting the number of records that were inserted into the database.
• Obtaining a step-by-step loading log, including error messages.
• Understanding performance metrics such as load time.

Alternatively, Transformer provides a ValidateLoad() method. You can use this method to validate that the schema of the XML file you are loading.

//validate and load an event
var loadInfo = myEvent.ValidateLoad();

This is especially helpful if you built your XML files from scratch and would like to validate the schema of your entire XML file before sending it to the database.

You can also call the Validate() method on an event to validate it's XML schema without loading it.

var validateInfo = myEvent.Validate() 

Verify Load Success

When you load an event, the Doodlebase API responds with information that can help you understand the end result of the load. You can work with this information to handle network connectivity issues, verify the format of your XML events, and more.

Here is an example of a code block saves a package to a local directory if the load is unsuccessful.

//Load package
var loadInfo = pkg.Load();

//Check load success
string storageDirectory = "C:\eventstorage"; 
if (loadInfo.Result == false)
{
    pkg.WriteZipFile(localStoragePath);
    Console.WriteLine("Load error");
}
else
{
    Console.WriteLine("Load success!");
}

This is a simplistic example. Your own program will likely contain additional load verification logic.

Next Steps

Congratulations! You now understand the basic structure of a data loading program. To understand the different properties of elements listed above, visit the Load > Event Reference page of the Doodlebase Documentation.

If you are ready to move on, your data should now be loaded and you're ready to visit the View > Concepts section of the documentation.