Why Automating Pass-Through Mapping Cuts Costs and Time

AvatarOctober 29, 2015

Historically, pass-through mapping has been a manual process. But for companies with a large volume of data in diverse formats, manual pass-through mapping is a time-consuming, costly process that is also prone to errors. Consider, for example, an international retail chain with hundreds of stores worldwide and point of sale data saved in multiple servers. The company needs to integrate the data and create a central repository, but with close to 200 tables of data in various formats, it would take a small army of developers to map the data manually.

Issues like these make the manual approach to mapping a challenge for many enterprise-level businesses. Yet the large majority (by my own estimate, 90 percent) rely on manual mapping, regardless of the volume of data and timeline of the project. By automating the process, however, your company can achieve higher quality data in significantly less time and for less cost.

How Pass-Through Mapping Automation Works
While data integration tools do not offer a simple method to automate pass-through mapping, automation is possible. It requires a deep understanding of the operating principles of the data integration tool, along with the ability to understand and replicate the XML code that the tool creates. Through code replication, one can develop a code generating utility within the tool.

How does this work? The code generation utility created through this process facilitates the synchronization of XML code and can generate the XML code for each table quickly — typically in one-tenth or less of the time it takes to do manually. Additionally, the code allows for a uniform style throughout the mapping process, eliminating human errors, such as forgetting to map a certain row or column.

Finally, because most of the leading data integration tools, such as Informatica, DataStage, and Talend, operate using metadata, the same automation process can be adopted and deployed across a large number of data integration tools and for a variety of data mapping processes (not just pass-through mapping).

Benefits of Automation
Automating the pass-through mapping process offers a number of benefits and eliminates many of the challenges of manual mapping. These include:

  • Time savings. Manual efforts, particularly for large volume or time-sensitive pass-through mapping projects, can require significant resources and manpower to complete in a timely manner. It simply may not be possible to “scale up” temporary resources for these types of projects. And even if it is, the management challenges of overseeing a large team of temporary developers may prove unruly.
    Done manually, a single mapping takes roughly four hours to create. With automation, however, the process takes around 5 minutes, meaning you could end up creating 100 mappings in a day, once the automation principle is in place.
  • Cost savings. The costs of large-scale mapping projects run high and may even be prohibitive. Say, for instance, you need to create 200 mappings in a short time period. Since each mapping requires four hours of a developer’s time, you would need to pay a team of developers for 800 hours to complete the project. On top of developer compensation, you would need to factor in recruiting, hiring, and management costs — all of which add up.
    Pre-made automation tools do exist, but they’re also expensive and still require a developer to run the process. Automation within the data integration tool itself, as described above, can therefore provide significant cost savings.
  • Better data. With manual mapping, if someone misses a column or imports a table and accidentally gives it the wrong name (e.g., T3495 instead of T394), significant consequences will ensue. Automation ensures uniform processes, which eliminates human error and produces higher quality data. Additionally, companies can avoid spending the time or resources required to find and fix human errors.

Automating pass-through mapping within the data integration tool isn’t easy and comes with plenty of challenges. It’s also a new endeavor that most developers are still struggling to figure out and successfully achieve. But for companies with large volume or time-sensitive pass-through mapping projects, it’s by far your best, most cost-efficient bet.

At InfoCepts, we’ve developed our own code to automate pass-through mapping — and could help you, too. Reach out to learn how.