DX Application Performance Management

Hi all folks,

we get lots of ERROR messages from BizTrxDef in the IntroscopeAgent.log and I don't know what's going wrong.

We use Introscope 9.5.2 and 9.5.5 with Websphere 7.0.0, Java 1.6 and Websphere 8.5.5, Java 1.6 and 1.7.

I disable biz-trx-http.pbd in the websphere-typical.pbl, the bizrecording.pbd is disabled by default.

websphere-typical.pbl

#biz-trx-http.pbd <- biz-trx-http.pbd

#bizrecording.pbd

#ServletHeaderDecorator.pbd

In the IntroscopeAgent.profile the bizRecording is set to false by default.

IntroscopeAgent.profile

introscope.agent.bizRecording.enabled=false

But I get all the time some messages like this in the IntroscopeAgent.log.

7/24/15 02:09:25 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/24/15 02:14:25 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/24/15 02:19:25 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/24/15 02:24:26 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

Any idea, what's going wrong or how to disable the BizTrxDef, in general every thing works well, nobody missed some data.

The Triage Map and Tess are not used and there is no database connect to the EM.

IntroscopeEnterpriseManager.properties

introscope.apm.feature.enabled=false

introscope.enterprisemanager.tess.enabled=false

With regards,

have a nice weekend,

Lutz

Hi Lutz,

my guess is that the message might occur because you disabled everything and not all code that relates to biz recording checks all those flags

You can try to suppress the errors written to the log if you set log4j.logger.IntroscopeAgent.BizTrxDef=FATAL in IntroscopeAgent.properties.

Or you can open a support case unless KulbirNijjer or DavidLewis jump in here.

Ciao,

Guenter

Here I go jumping in..

I don't want this to sound like a knee-jerk reaction but it's preferable to get a DEBUG log for this to generate stack traces so we would know why the error is appearing in particular as I think that same error can have multiple root causes.

You can do selective DEBUG log4j.logger.IntroscopeAgent.BizTrxDef=DEBUG, logfile

so that the whole log is not full of other things..

My feeling is the same as Guenter's, in spite of the efforts to supress issues related to the database (particularly because there isn't one), it's going to link back to the database being missing.

We can look at lots of ways of supressing the errors but ideally I'd like to confirm a root cause which is why this is what I would suggest next.

Hi David,

this is a good idea, I try to collect some more data to find out what's going wrong.

I'll be back,

Lutz

Hi David,

I add log4j.logger.IntroscopeAgent.BizTrxDef=DEBUG, logfile to the IntroscopeAgent.profile and now I got the following additional messages.

7/30/15 08:21:47 PM CEST [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.bizdef.turnOff.nonIdentifying.txn) set to default value of "true"

7/30/15 08:21:47 PM CEST [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.bizdef.matchPost.vetoedURI.max) set to default value of "1000"

7/30/15 08:21:52 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/30/15 08:21:52 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/30/15 08:21:52 PM CEST [DEBUG] [IntroscopeAgent.BizTrxDef] Unable to register Business Definitions: Invalid Registry Path Exception

com.wily.isengard.messageprimitives.service.ServiceNotFoundException: Invalid Registry Path Exception

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:347)

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:253)

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.getService(MessageServiceFactory.java:144)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator.loadLibrary(BizDefAdministrator.java:979)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator.registerAgentToEM(BizDefAdministrator.java:363)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator$1$1.connectionUp(BizDefAdministrator.java:298)

        at com.wily.introscope.agent.connection.IsengardServerConnectionManager.addConnectionObserver(IsengardServerConnectionManager.java:455)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator$1.ITimestampedRunnable_execute(BizDefAdministrator.java:291)

        at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)

        at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)

        at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)

        at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)

        at java.lang.Thread.run(Thread.java:795)

Caused by:

com.wily.isengard.util.tree.EntryNotFoundException for entry: /service/com.wily.apm.spec.tess.beans.bizdefinition.IBizDefinitionLibrary

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:348)

        ... 12 more

7/30/15 08:21:52 PM CEST [DEBUG] [IntroscopeAgent.BizTrxDef] Unable to register Business Definitions: Invalid Registry Path Exception

com.wily.isengard.messageprimitives.service.ServiceNotFoundException: Invalid Registry Path Exception

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:347)

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:253)

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.getService(MessageServiceFactory.java:144)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator.loadLibrary(BizDefAdministrator.java:979)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator.registerAgentToEM(BizDefAdministrator.java:363)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator$1$1.connectionUp(BizDefAdministrator.java:298)

        at com.wily.introscope.agent.connection.IsengardServerConnectionManager.addConnectionObserver(IsengardServerConnectionManager.java:455)

        at com.wily.introscope.agent.bizdef.BizDefAdministrator$1.ITimestampedRunnable_execute(BizDefAdministrator.java:291)

        at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)

        at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)

        at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)

        at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)

        at java.lang.Thread.run(Thread.java:795)

Caused by:

com.wily.isengard.util.tree.EntryNotFoundException for entry: /service/com.wily.apm.spec.tess.beans.bizdefinition.IBizDefinitionLibrary

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:348)

        ... 12 more

7/30/15 08:24:38 PM CEST [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Executing Bootstrap Classes Dynamic Instrumentation.

7/30/15 08:24:38 PM CEST [INFO] [IntroscopeAgent.BootstrapInstrumentationManager] Processed All Bootstrap Classes.

Maybe, some know what's going wrong.

Lutz

HI Lutz,

I think this proves that the ultimate issue, is still, unfortunately, the database.

com.wily.isengard.util.tree.EntryNotFoundException for entry: /service/com.wily.apm.spec.tess.beans.bizdefinition.IBizDefinitionLibrary

        at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:348)

There would be particular services running on the Enterprise Manager when the database is there/available, and when the database is not present, the services will not initialize, and the agent throws an error because the services are not available.

That's on a simple level anyway - the business definitions service/process in agent will not register because it doesn't have anything to register with on the EM.

Being one of the voices of support (of which there are a few on this discussion), it isn't supported to run without the database and we continue to rely on it for new features I know that things like APM Team Center (newest version of Triage Map) will not operate without it.

I think customer ideas have been raised to request a version of APM that doesn't need this database but I simply don't know if this will be implemented,

For Postgres on AIX, I think it's difficult to find a pre-compiled package of Postgres at all, particularly one that installs/runs happily on AIX, this is one reason it is difficult to support it.

Sorry we can't suggest anything more constructive for you.

David

Hi David,

thanks for your useful explanation.

For Postgres on AIX, I think it's difficult to find a pre-compiled package of Postgres at all, particularly one that installs/runs happily on AIX, this is one reason it is difficult to support it. 

This is not a problem in general, we use Postgres 9.3.x on AIX in general. And I use Postgres 9.3.5 on AIX 7.1 with Introscope, but this is not supported by CA, not a fix for the problem also.

The next time a CA Salesdroid talk to us to sell new products, we talk to him about useful products.

I think customer ideas have been raised to request a version of APM that doesn't need this database but I simply don't know if this will be implemented,

As long as we have no benefit, I spend no time to enable the Triage Map or a useless DB.

Regards, Lutz

p.s.

Sorry, maybe I'm wrong.

But today the Triage Map is useless for us. And I don't know any other reason to install a additional DB.

Hi David/Lutz,

Just an FYI

When I recreated the problem in my update below (https://communities.ca.com/message/241812152#241812152) I still had the DB offline after I removed these 2 properties which fixed the problem:

introscope.apm.feature.enabled=false

introscope.enterprisemanager.tess.enabled=false

So I think they are controlling the EM services availability rather than the DB accessibility itself.

Regards,

Lynn

The error occurs when the agents aren't able to get the business definitions from their respective collectors.

Possible reasons:

1) Slowness in the cluster.

2) Slowness in DB.

If the definitions in the collector are not able to make it to the agents within 15 secs, you will encounter this error. if unsuccessful the agents retires it every 5 mins.

Please check the following.

1) Number of components defined

2) postgres/oracle db logs when you saw the problem

3) Check if you are seeing any cluster slowness(in terms of agents connecting to EM).

4) Otherwise, briefly turn on the agent in debug mode. 15 mins is fine as the error occurs every 5 mins. and contact ca support.

Hi James,

as states above:

introscope.apm.feature.enabled=false

introscope.enterprisemanager.tess.enabled=false

Ciao,

Guenter

To disable the CEM integration which should stop the error messages I would suggest:

• Remove the 2 jar files BizTrxHttp.jar & ServletHeaderDecorator.jar from the AGENT_HOME/core/etc directory
• Set this property to false if it set to true: introscope.agent.decorator.enabled=false
• Restart the agent

Removing both Jar's won't fix anything on that side.

BT's are getting permanently pushed down to the agents. It doesn't matter whether the Business Transaction discovery or even the header decoration is enabled on the agent.

Lutz, can you please try to get for us a few lines of log with this ERROR in DEBUG mode. Switching log levels is a hot property.

Hi Lynn,

this is the default

introscope.agent.decorator.enabled=false

and it is set to false. But I feel bad to delete the jar files.

Bye, Lutz

Hi Lutz,

"The Triage Map and Tess are not used and there is no database connect to the EM.

IntroscopeEnterpriseManager.properties

introscope.apm.feature.enabled=false

introscope.enterprisemanager.tess.enabled=false

"

This is not even a supported configuration. We don't support\test APM environment without APM database configured and regarding the second property for TESS was this provided by CA Support for troubleshooting a issue ? If not then kindly revert it back to true regardless if you use CEM in the environment or not.

Regards,

Kulbir.

Hi Kulbir,

it was suggested by CA, because we are not interesting in using the Triage Map.

This is not even a supported configuration. We don't support\test APM environment without APM database configured and regarding the second property for TESS was this provided by CA Support for troubleshooting a issue ? If not then kindly revert it back to true regardless if you use CEM in the environment or not.

Today we use the Triage Map in a test and an evaluation environment only, but there is no benefit for us.

In general we use Introscope 9.5.2 like Introscope 8.2.3 and nobody plan a Triage Map rollout for the future.

Sorry,

Lutz

Hi Lutz,

Disabling Triage Map is fine and supported but not disabling TESS and running APM environment without APM database.

Problems you see on Agent are very likely a side effect of doing same and a good check would be to replicate same in a test environment and validate.

Regards,

Kulbir.

CA Support.

Hi Lutz,

Supportability issues aside ,I tried to setup a somewhat similar environment

-EM pointed to dummy DB, TESS disabled

-Pointed 9.5 Agent to EM

however didn't see similar error. So it will be good if you can enable debug logging on Agent and capture more details on error for us to look at.

Regards,

Kulbir.

Kulbir,

what if we defined Business Transaction mapping on that environment for the Agent side based on URL or Parameters. Could this turn side ways...

Hi Kulbir/Andreas,

The background on this environment may not be 100% relevant but I managed to recreate the ERROR message on an existing 9.6 weblogic/medrec agent that had CEM integration previously enabled through a 9.6/9.7.1 EM.

I did not start the DB & I modified the files as follows for a EM & Agent restart

IntroscopeEnterpriseManager.properties:

Added

introscope.apm.feature.enabled=false

introscope.enterprisemanager.tess.enabled=false

IntroscopeAgent.profile:

Disabled:

introscope.agent.bizRecording.enabled=false

introscope.agent.decorator.enabled=false

introscope.agent.appmap.enabled=false

Removed CEM related pbds:

# introscope.autoprobe.directivesFile=weblogic-full.pbl,spm.pbl,bizrecording.pbd,servletheaderdecorator.pbd,hotdeploy

introscope.autoprobe.directivesFile=weblogic-typical.pbl,spm.pbl,hotdeploy

The message occurs every 5 minutes & with DEBUG enabled shows:

7/30/15 11:13:28 AM EST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/30/15 11:13:28 AM EST [DEBUG] [IntroscopeAgent.BizTrxDef] Unable to register Business Definitions: Invalid Registry Path Exception

com.wily.isengard.messageprimitives.service.ServiceNotFoundException: Invalid Registry Path Exception

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:347)

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:253)

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.getService(MessageServiceFactory.java:144)

at com.wily.introscope.agent.bizdef.BizDefAdministrator.loadLibrary(BizDefAdministrator.java:979)

at com.wily.introscope.agent.bizdef.BizDefAdministrator.registerAgentToEM(BizDefAdministrator.java:363)

at com.wily.introscope.agent.bizdef.BizDefAdministrator$1.ITimestampedRunnable_execute(BizDefAdministrator.java:337)

at com.wily.util.heartbeat.IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972)

at com.wily.util.heartbeat.IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491)

at com.wily.util.heartbeat.IntervalHeartbeat.access$2(IntervalHeartbeat.java:445)

at com.wily.util.heartbeat.IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)

at java.lang.Thread.run(Thread.java:619)

Caused by: com.wily.isengard.util.tree.EntryNotFoundException for entry: /service/com.wily.apm.spec.tess.beans.bizdefinition.IBizDefinitionLibrary

at com.wily.isengard.messageprimitives.service.MessageServiceFactory.internalGetServiceInterface(MessageServiceFactory.java:348)

... 10 more

As a test I did another Agent restart with the BizTrxHttp.jar removed which did not prevent the ERROR as Andreas suggested..

With BizDef.jar also removed it stopped the ERROR but then not surprisingly caused new ones with ("Exception started method tracer...")

After a complete restart with above 2 jars restored and these EM properties removed the ERROR then stopped:

introscope.apm.feature.enabled=false

introscope.enterprisemanager.tess.enabled=false

Regards,

Lynn

Thanks Lynn this is helpful.

Lutz so to sum up this lengthy discussion please do following so your environment is first supported:

-Point EM to a valid APM Database(Postgres or Oracle)

-Enable TESS (Property to disable only exists for troubleshooting purposes by support and unless directed should not be specified or set to false) regardless whether you use CEM or not.

Remember you can do Agent only recording to define Business Transactions(with Tess enabled) without needing any TIM deployment, for more details refer to APM Documentation.

Once above is done and Agent still reports this error open a support issue with CA, also kindly enable debug logging on Agent and attach both EM\Agent logs.

Regards,

Kulbir.

Hi Kulbir,

what is the benefit of a additional APM Database without using Triage Map and/or TESS.

-Point EM to a valid APM Database(Postgres or Oracle)

-Enable TESS (Property to disable only exists for troubleshooting purposes by support and unless directed should not be specified or set to false) regardless whether you use CEM or not.

I try to do this in the test environment, but I spend no time to install useless databases to the systems to suppress a message. If this is the only way to clean my logs, I follow Guenter by setting log4j.logger.IntroscopeAgent.BizTrxDef=FATAL to suppress the messages.

On the other hand, I'm on AIX.

But CA does not support Introscope with Postgres on AIX and Oracle is not an option for some/the most of my Introscope EM systems.

With regards, Lutz

Hi Lutz,

David's response pretty much summarizes this issue and w.r.t your concern about Triage Map not providing any useful information, perhaps you need to look into changes we did in APM 10 with Team Center. It's a complete redesign of Appmap + much more, features far more functionality and greater flexibility then anything you have seen in the past and we plan to enhance it further going forward.

I will recommend you to attend this upcoming webcast on APM 10 features and I am sure your opinion on usefulness of APM database will change.

>>>>>>>>>>>>

Want to learn more about latest APM 10 release and why it matters ?

Tune in to upcoming webinar on August 18th 9am PST by CA APM Product Management team:

https://www.brighttalk.com/webcast/7819/161189?=c

>>>>>>>>>>>>

With respect to Postgres on AIX currently it's not supported but you can definitely file a idea so Product Management can review it for future releases or else deploy Postgres on a remote Linux\Windows\Solaris environment and point EM to same.

Hope this helps.

Regards,

Kulbir.

Hi Kulbir,

if a recorded version of the webcast became available, I have a look to the news/the webcast.

But in general, I can't see any must-have features became available with APM 10. And some of the bugs became available with APM 9.7.0/1 are still available with APM 10.0.0.

But next year, I plan to migrate to a new Introscope APM version, at the time Introscope 9.5.2 is mine.

Regards, Lutz

Hi Kulbir,

I do the testing, based on Introscope 9.5.2 with Postgres 9.3.5 on AIX 7.1.

I got the error Messages all the time.

7/31/15 06:31:47 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/31/15 06:36:46 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/31/15 06:41:46 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/31/15 06:43:08 PM CEST [ERROR] [IntroscopeAgent.IsengardServerConnectionManager] Agent reporter thread caught unexpected exception.

7/31/15 06:43:31 PM CEST [WARN] [IntroscopeAgent.ConnectionThread] Failed to re-connect to the Introscope Enterprise Manager at d100atu61401.d100.intern:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory (1).

7/31/15 06:43:31 PM CEST [WARN] [IntroscopeAgent.ConnectionThread] Failed to re-connect to the Introscope Enterprise Manager at d100atu61401:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory (1).

7/31/15 06:43:31 PM CEST [INFO] [IntroscopeAgent.Agent] ThreadDumpGenerator is enabled

I start a PG DB and restart the EMs with "introscope.apm.feature.enabled=false" and "introscope.enterprisemanager.tess.enabled=true"

7/31/15 06:46:53 PM CEST [INFO] [IntroscopeAgent.Instrumentation Manager] Agent Dynamic Instrumentation bean has been deployed.

7/31/15 06:46:53 PM CEST [INFO] [IntroscopeAgent.IsengardServerConnectionManager] Connected controllable Agent to the Introscope Enterprise Manager at d100atu61401.d100.intern:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory. Host = "D100SEUSBB31", Process = "WebSphere", Agent Name = "V32E.SB0.jms.a.1.as91051", Active = "true".

7/31/15 06:46:53 PM CEST [INFO] [IntroscopeAgent.ConnectionThread] Connected to d100atu61401.d100.intern:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory in allowed mode.

The Agent reconnect to an EM and I got no new error messages.

A short time later, I restart the EMs with "introscope.enterprisemanager.tess.enabled=false" again.

7/31/15 07:01:01 PM CEST [WARN] [IntroscopeAgent.ConnectionThread] Failed to re-connect to the Introscope Enterprise Manager at d100atu61401.d100.intern:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory (1).

7/31/15 07:01:01 PM CEST [WARN] [IntroscopeAgent.ConnectionThread] Failed to re-connect to the Introscope Enterprise Manager at d100atu61401:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory (1).

7/31/15 07:01:01 PM CEST [INFO] [IntroscopeAgent.Agent] ThreadDumpGenerator is enabled

7/31/15 07:01:01 PM CEST [INFO] [IntroscopeAgent.Instrumentation Manager] Agent Dynamic Instrumentation bean has been deployed.

And the error messages are back after the agent reconnect to an EM again.

7/31/15 07:01:01 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/31/15 07:01:01 PM CEST [INFO] [IntroscopeAgent.IsengardServerConnectionManager] Connected controllable Agent to the Introscope Enterprise Manager at D100ATU61399.D100.intern:5001,com.wily.isengard.postofficehub.link.net.DefaultSocketFactory. Host = "D100SEUSBB31", Process = "WebSphere", Agent Name = "V32E.SB0.jms.a.1.as91051", Active = "false".

7/31/15 07:01:01 PM CEST [WARN] [IntroscopeAgent.IsengardServerConnectionManager] Lost contact with the Introscope Enterprise Manager at D100ATU61399.D100.intern:5001.

7/31/15 07:01:02 PM CEST [INFO] [IntroscopeAgent.Agent] Transaction Tracer sampling interval has been set to 120 seconds.

7/31/15 07:01:02 PM CEST [INFO] [IntroscopeAgent.Agent] Transaction Tracer samples per interval has been set to 1.

7/31/15 07:01:03 PM CEST [INFO] [IntroscopeAgent.Agent] ThreadDumpGenerator is enabled

Now I get the know error messages all the time, again.

7/31/15 07:01:41 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/31/15 07:01:51 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

7/31/15 07:06:46 PM CEST [ERROR] [IntroscopeAgent.BizTrxDef] Error occurred and could not register Business Definitions

Quod erat demonstrandum, you are right Kulbir.

I fix the problem by the way Guenter suggested,

regards, Lutz