Installing the TIBCO® Metadata Agent
Related information
Installation and configuration
TIBCO® Metadata agent is a Java application provided as a jar file and configured using a properties file. Its main purpose is to establish interactions between the source system instance (through specific adapter) installed on a server and the EBX® Metadata Management application. One agent can be connected to one instance of a source system. The EBX® Metadata Management application supports connections to multiple agents that manage different data sources.
An agent can have one or multiple adapters to support metadata extraction from different data sources. An adapter is a separate Java application that ensures one or several of the metadata capabilities for a given data source type(s):
-
Data source management
-
Metadata harvesting
-
Data sampling
-
Metadata provisioning
-
Lineage extraction.Prior to agent configuration, you must create a technical user from the Administration perspective. For more details, see Technical Users.Note: the technical user created is automatically prefixed with ebx-.
Prerequisites
The TIBCO® Metadata Agent requires:
-
JRE 11 minimumIf the TIBCO® TDV Metadata Adapter is used (see TIBCO TDV Metadata Adapter installation):
-
TIBCO® Data Virtualization version 8.3 minimum
Security and Cloud Access
In order to access to EBX® (CIC Cloud), and to allow the EBX® Metadata Management application to access to the TIBCO® Metadata Agent, you must take into account the following infrastructure considerations:
-
TIBCO® Metadata Agent hosting server must be opened to the Internet.
-
It is recommended to define a DNS Name for the TIBCO® Metadata Agent hosting server, in order to ease certificate management.
-
TIBCO® Metadata Agent hosting server must be reachable via HTTPs (HTTP over SSL), using a CA-signed certificate (no self-signed certificate).
-
TIBCO® Metadata Agent external port must be port 443 (default HTTPs port), using a reverse-proxy (IIS, Apache, Ngninx, …). (ma.http.ws-url has to be set to the reverse-proxied URL to the agent, reachable from Internet).
-
TIBCO® Metadata Agent internal port can be configured on any available port on the machine (see ma.http.port in Agent configuration properties), and use in the reverse-proxy configuration to expose the TIBCO® Metadata Agent to Internet.
Installation process
The following steps describe how to get the TIBCO® Metadata Agent running on your system:
-
Download TIBCO® Metadata Agent package from the Metadata Administration perspective.
-
Unpack it on your system.
-
Fill in properties template as described below in the Properties configuration section. Please note that for some properties it is required to log into EBX® and/or obtain some information from a EBX® administrator.
-
Download and install the EBX® Metadata Management application Adapters required for your source systems.
-
Fill in properties template for each adapter.
-
Activate required adapters in the main Agent properties template.
-
Launch TIBCO® Metadata Agent as described below in the Execution section.
Packaging
The agent is available as a standalone Java application (jar file). You can download it from the Metadata Administration perspective.
Properties files
The main packaging of the TIBCO® Metadata Agent includes following properties files:
-
application-metadata-agent.properties for configuration of Agent
-
application-datasources.properties for configuration of the supported data source types per adapter
-
application-tdv.properties (optional) for configuration of the TIBCO® TDV Metadata Adapter.
TIBCO® Metadata Agent properties template
######################################################
######################################################
##### ##### ##### Metadata connection ##### ##### #####
######################################################
######################################################
ma.metadata.context=[[METADATA BASE URL]]
ma.metadata.username=[[METADATA TECHNICAL USER NAME]]
ma.metadata.password=[[METADATA TECHNICAL USER PASSWORD MAY BE ENCRYPTED]]
ma.metadata.public-key=[[METADATA TECHNICAL USER JWT]]
ma.metadata.label=[[LABEL FOR THIS AGENT]]
####
#### Uncomment this line if you want to use encryption for agent passwords.
#### In this case, you will have to encrypt password using TIBCO Metadata Agent Tools
#### Read the documentation for more information
####
#ma.metadata.useEncryption=true
######################################################
######################################################
##### ##### ##### Agent properties ##### ##### #####
######################################################
######################################################
ma.http.port=[[AGENT PORT]]
ma.http.ws-url=[[AGENT CONTEXT URL]]
######################################################
######################################################
##### ##### ##### Agent log properties ##### ##### #####
######################################################
######################################################
#Folder where the logs will be stored
ma.logging.path=./logs
#Log level for technical logs (Spring framework, external libraries).
ma.logging.level.root=INFO
#Log level for the Agent's functional logs.
ma.logging.level.agent=INFO
#Log level for the SOAP messages. Set to TRACE to see full request body.
ma.logging.level.soap.sent=INFO
#Log level for the SOAP messages. Set to TRACE to see full response body.
ma.logging.level.soap.received=INFO
######################################################
######################################################
##### ##### ##### Agent proxy properties ##### ##### #####
######################################################
######################################################
####
#### Uncomment this line if you want to use a proxy configuration for the agent
####
#ma.proxy.useProxy=true ma.proxy.proxyHost=[[PROXY HOST]]
ma.proxy.proxyPort=[[PROXY PORT]]
ma.proxy.proxyAuthentication=[[PROXY
AUTHENTICATION ?]]
ma.proxy.proxyUsername=[[PROXY USERNAME]]
ma.proxy.proxyPassword=[[PROXY PASSWORD]]
######################################################
######################################################
##### ##### ##### Agent Adapter properties ##### ##### #####
######################################################
######################################################
#Comma-separated list of packages to scan for adapters classes
agent-adapter.base.package=com.tibco.ca.tabula.agent.tdv
#Comma separated list of profiles (corresponding to adapter technical names)
agent-adapter.active-profiles=tdv
Properties configuration
Prior to launching the TIBCO® Metadata Agent, you need to complete the information in the properties file to check the connection to the EBX® Metadata Management application.
Users are strongly discouraged to modify properties that are not described in the table below but are present in the properties file, especially specific URLs configured for EBX® Metadata Management application operation or the H2 database JDBC URL. A good practice is to remove old specific properties from your prior-to-version 4 agent, in order to keep only specific properties in your configuration file.
| Property name | Description |
|---|---|
| TIBCO EBX®'s metadata connection | |
ma.metadata.contex t |
REST Endpoint for EBX® Metadata Management application Advanced Data Services. Expected format is: https://:/ |
ma.metadata.username |
Metadata technical user name. Please, contact an administrator to receive this information. |
ma.metadata.password |
Metadata technical user password. Please, contact an administrator to receive this information. |
ma.metadata.public-key |
Public key (JWT) associated with the EBX® Metadata Management application technical user. Please, contact an administrator to receive this information. |
ma.metadata.label |
A user-friendly label to show in the Metadata for this agent. |
ma.metadata.useEncryption |
Define if encryption is used for ma.tdv.datasource.password, ma.tdv.soap.password, ma.metadata.password and ma.proxy.proxyPassword. If activated, all passwords must be encrypted using TIBCO Metadata Agent Tools, see below. |
| Agent properties | |
ma.http.port |
The port on which the agent listens for incoming REST connections. |
ma.http.ws-ur l |
The URL of the agent's REST endpoint. You have to set HTTPs on the agent (using reverse proxy for instance, or the EBX® Metadata Management application won't be able to connect).You also have to allow incoming connections from cloud (Internet) to agent's host & port. |
| Agent logging properties | |
ma.logging.path |
The folder where the agent's logs will be stored. Agent must have write permission on this folder. |
ma.logging.level.root |
Log level for the technical logs generated by the technical framework that the Agent runs on. |
ma.logging.level.agent |
Log level for the functional logs generated by the Agent. |
ma.logging.level.soap.sent |
Log level for the SOAP messages that are sent to the TDV from the Agent. By default, they are not logged. If you want to log them for debug or any other purposes, set this property to TRACE. |
ma.logging.level.soap.received |
Log level for the SOAP messages that are received from the TDV to the Agent. By default, they are not logged. If you want to log them for debug or any other purposes, set this property to TRACE. |
| Agent proxy properties | |
ma.proxy.useProxy |
Uncomment this line if you want to use an HTTP(s) proxy for the agent. |
ma.proxy.proxyHost |
Proxy host. |
ma.proxy.proxyPort |
Proxy port |
ma.proxy.proxyAuthentication |
If authentication is needed for proxy usage, set this property to true. |
ma.proxy.proxyUsername |
Username to be used for the proxy. |
ma.proxy.proxyPassword |
Password to be used for the proxy. |
| Agent adapter properties | |
agent-adapter.base.package |
Comma-separated list of packages to scan for adapters classes. |
agent-adapter.active-profiles |
Comma separated list of profiles (corresponding to adapter technical names). |
Once these properties are configured, you can launch the agent as a Java application, using command line or any other suitable tool (see below on this page).
The configuration file should be named
application.properties.
Upon the first launch, the agent will query the EBX® Metadata Management application REST API for registration and will obtain a GUID to be used in all information exchanges between the two applications. Check that in the agent’s log file after startup you see the message "Started MetadataAgentApplication in xx seconds." This indicates that the start up was correct and the agent is running. If you don’t see this message, your need to check the log file for error messages explaining the issue preventing agent to start.
Once registered, the agent will be displayed in the EBX® Metadata Administration perspective as described TIBCO® Metadata Agents.
Data source types properties file
The properties file application-datasources.properties allows the agent to make connection between supported data source types, metadata management functions and the adapters to use.
Properties in this file must follow the pattern: datasource.operation\_name.data\_source\_type=adapter\_profile Supported operation names are as follows:
-
dsmanagement
-
harvesting
-
sampling
-
provisioning
-
lineage.
Data source type name must be formatted as alphanumeric name all in lower case without any special characters. You can also define a label for data source type that will be displayed in the EBX® Metadata Management application. Only one adapter can be defined for operation - data source type.
Below is the configuration pattern:
datasource.label.<data source type>=
datasource.dsmanagement.<data source type>=
datasource.harvesting.<data source type>=
datasource.sampling.<data source type>=
datasource.provisioning.<data source type>=
datasource.lineage.<data source type>=
By default this file includes configuration for the data source types supported by TIBCO TDV Metadata Adapter installation.
EBX® Metadata Management application Connection Encryption
If
ma.metadata.useEncryption is activated, then ma.tdv.datasource.password, ma.tdv.soap.password, ma.metadata.password and ma.proxy.proxyPassword must be encrypted into application.properties before starting the agent:
-
As a prerequisite,
ma.metadata.public-keymust be defined inapplication.propertiesaccording to the corresponding EBX® Metadata Management application Technical User. -
Use TIBCO® Metadata Agent Tools to encrypt the original password. Launch in a terminal:
java -jar metadata-agent-tools.jar MY_PLAIN_PASSWORDSee the output sample below. -
Copy-paste the encrypted values into
application.propertiesfor each corresponding password property.
The output provided is as follows:
`**************************`
`Value to encrypt is MY_PLAIN_PASSWORD`
`**************************`
`Encrypted value is : I7UOEPOHtF+J4EYPTPRxR0++QG4hoXFe87uSPluI1lbi`
`**************************`
Caution: with this feature enabled, an encrypted password must be re-encrypted on each renewal of JWT key.
Agent execution (as a Standalone Java Application)
You have to configure a launcher to execute the agent, adding the configuration files into a config directory under the directory where is placed the agent.
logs and storage directories are created by the agent itself directly under the structure.
Under storage, two files are created, one for agent local database (configDB.mv.db), one for scheduling ( quartzDB.mv.db). If you have previously used different names for these files in your previous configuration, please rename these files accordingly in your file system, after stopping the current agent.
To launch the agent, you can create a shell file as follows (in this example the jars are placed in a /metadata-agent directory):
You have to define the default encoding as UTF-8 for the JVM using
-Dfile.encoding=UTF-8, if not already set.
Example on Linux (with TDV adapter activated):
java -Dloader.path=metadata-agent-tdv-adapter.jar,metadata-agent-tdv-driver-jdbc.jar -Dspring.profiles.active=tdv -jar metadata-agent.jar >> logs/metadata-agent.log 2>&1 &
Example on Windows (with TDV adapter activated):
java -Dfile.encoding=UTF-8 -Dloader.path=metadata-agent-tdv-adapter.jar,metadata-agent-tdv-driver-jdbc.jar -Dspring.profiles.active=tdv -jar metadata-agent.jar > /logs/metadata-agent.log 2>&1
Once started, an
application.pid file is created containing the process id of the agent. You can stop the agent by killing this process. kill $(cat application.pid)
The agent can be also shut down using a HTTP POST request on a dedicated endpoint:
curl -X POST http://<agent_host>:<agent-port>/manage/shutdown
Troubleshooting
The following table provides causes and resolutions to two possible errors: