Usercube-Prepare-Synchronization

Usercube-Prepare-Synchronization is used as the second step of the Synchronization process. It cleanses exported CSV files before sending them to the server for database loading. It is performed on the Agent side.

Behavior Details

The task reads files from the source directory, usually the temp folder > ExportOutput folder. See the Application Settings topic for additional information.

Cleanse data

The following actions are performed on the CSV source files:

1. Remove columns that are not used in Entity Type Mapping or Entity Association Mapping .
2. Remove entries that have a null primary key.
3. Remove duplicates.
4. Sort entries according to the primary key.

The result of the Prepare-Synchronization is stored in the Application Settings as three files:

• For every entity type of the relevant Connector involved in an Entity Type Mapping or an Entity Association Mapping , a .sorted.csv file is generated, containing the final, cleansed and sorted result.
• Duplicates are kept in a separate .duplicates.csv file.
• Null primary key entries are kept in a separate .nullpk.csv file.

All files produced by the task are in the work folder > Collect directory. See the Application Settings topic for additional information.

Compute changes

In incremental mode, changes might need to be computed by the Agent:

• If the Export step has provided computed changes, no further process is required. The changes will be sent as-is to the server.
• If the Export step has provided a full extract of the managed systems, the Prepare-Synchronization step computes changes. This computation is based on the result of the last data cleansing, generated by the previous Prepare-Synchronization, and stored in the previous folder in the export directory. See the Application Settings topic for additional information.

For incremental mode, it is recommended, whenever possible, to use managed systems to compute changes. Dedicated workstations and knowledge of the inner data organization allow managed systems to compute changes with performance that Identity Manager can't match. Using managed systems for these operations avoids generating heavy files and alleviates Identity Manager's processing load.

The result is a set of clean lists of changes stored as a .sorted.delta file containing a command column. The command column can take the following values:

• insert
• update
• delete
• merge

These values are instructions for the Synchronization step to apply the changes to the database.

The .sorted file (that is, the original clean export file, not the changes) is stored in the previous folder inside the export directory. It will be used as a reference for the next incremental Prepare-Synchronization to compute the changes, if needed. See the Application Settings topic for additional information.

Tampering with the previous folder content would result in false changes leading to false computation. It would result in data corruption in the Identity Manager database. To restore the Identity Manager database and reflect the managed system data updates, a complete__Sync Up would be required.

Prepare the server

At the beginning of every Prepare-Synchronization process, the Server is notified via HTTP.

Upon receiving the notification, the server creates a directory on its host environment, identified by a unique GUID, to contain .sorted or .sorted.delta files that will be sent by the agent.

This is designed to prevent network errors that would cause an incremental database update to happen more than once.

This means that several Export and Prepare-Synchronization tasks can be executed simultaneously. These tasks will be processed by the server one at a time, in the right order.

Any notification of a complete Prepare-Synchronization would cancel the previous non-processed incremental Prepare-Synchronizations. As a complete Prepare-Synchronization reloads the whole database, it renders incremental changes computation moot.

Send clean exports

.sorted or .sorted.delta files are sent over HTTP to the Server for the last step.

Example

The figure models the complete Prepare-Synchronization steps applied to an Active Directory export. The matching Connector defines an Entity Type AD Entry and two associations (members and manager).

Examples

Usercube-Prepare-Synchronization can be used as an executable file as follows:

./identitymanager-Prepare-Synchronization --api-url myserver.usercube.com --api-client-id myclientid --api-secret myclient secret --connector --agent myagent --synchronization-mode complete

Arguments

Name	Details
--agent required	Type Agent Description Identifier of the agent where the task runs.
--connector required	Type Connector Description Identifier of the linked connector. The task is linked to a connector whose entity types are synchronized.
--synchronization-mode required	Type Upward Data Synchronization Mode Description Synchronization mode for this task can be one of the following: - Initial - Complete - Incremental This must be the same as the associated Export and Synchronize tasks. Use initial if this is the first time the target managed system is synchronized. Use complete to load the data from the managed system as a whole. Use incremental to consider only incremental changes from the last synchronization. In incremental mode, the Prepare-Synchronization task computes changes in the source managed system since the last Prepare-Synchronization.
--sources-directory default value: ExportOutput	Type String Description Directory path, relative to temp folder, from which export files to cleanse are read. See the Application Settings topic for additional information
--working-directory default value: Collect	Type String Description The directory path, relative to work folder, to which intermediary and cleansed files are stored. See the Application Settings topic for additional information
---	---
--api-client-id required	Type String Description Login used to authenticate to the server. Every request from agent to server needs to be authenticated with an OpenID Connect ClientId/Secret pair, linked to a profile with the relevant permissions. See the OpenIdClient topic for additional information.
--api-secret required	Type String Description Password used to authenticate to the server. Every request from agent to server needs to be authenticated with an OpenID Connect ClientId/Secret pair, linked to a profile with the relevant permissions. See the OpenIdClient topic for additional information.
--api-url required	Type String Description URL of Identity Manager server.
---	---
--log-level optional	Type LogLevel Description Level of log information among: Verbose; Debug; Information; Warning; Error; Fatal.