Link back to index: https://community.sap.com/t5/technology-blogs-by-members/blog-series-on-my-cpi-camel-learning-journey/ba-p/14031053
Problem statement
Ok, in our previous demo we finally had to admit that we cannot jut use the blueprint CPI produces.
This is rather unfortunate because like I said in the video, it brings us back to this picture
Here we see that Green stuff is what we intend to control through test cases design and mocks/stubs.
But the most important part is Amber – our Code Under Test.
And of course, we are not allowed to modify our code in order to put it under test.
This is a big problem, because what that really means for us – is in all cases I showed earlier (and will continue showing), we were testing locally something different than original iflow.
I guess, I was referring to it as “local blueprint” (from blueprint-local.xml) while the original one (from OSGI-INFO/bluerint/beans.xml) – as “remote blueprint”.
Did it all make any sense then?
Well, maybe – if we can establish the equivalence between them in regards to behaviour we intend to test.
But why not simply test remote blueprint?
Remember the simple sender initiated scenario – we serve the http endpoint to reply with some stuff we fetch from remote odata service.
Not only that means we need to run the http server somehow (and dispatch inbound requests to it in cloud), but we also deal with authentication (remember that jwt “ESBMessaging.send” scope?)
And that tightly couples us to BTP and CPI runtime and environment (components they have there to interact with BTP).
A little bit of osgi finally
So, how does that work in OSGI world?
Well, bundles provide services to each other.
And so in its META-INF/MANIFEST.MF bundle can define what it imports, exports and provides.
This is essential for us (especially in terms of local testing) because this is also how our script bundles and message mapping bundles are discovered/linked by our iflow bundles (remotely) and blueprints (locally).
For example, this is how dependencies look like for our friend Basics_Exception_Subprocess on cpi and locally.
Just to show you that it is A LOT, so just running it locally would actually mean having the same “complete cloud environment” on your machine.
We use webshell to run “bundle:headers N” command to get this list, but as we have it deployed as bundle, I would expect it to exactly match manifest (to the right)
I apologise for how it looks, but code snippets suck so much here that eventually I just put stuff into the table (and at least it doesn’t add those bloody empy lines before and after)…
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Basics_Exception_Subprocess
Bundle-SymbolicName: Basics_Exception_Subprocess
Bundle-Version: 1.0.0
SAP-BundleType: IntegrationFlow
SAP-NodeType: IFLMAP
SAP-RuntimeProfile: iflmap
Import-Package: org.apache.camel.processor.errorhandler, com.sap.esb.ca
mel.http.ahc.configurer.retry;version=”[1,4)”, com.sap.esb.camel.http.a
hc.configurer;version=”[1,2)”, com.sap.it.nm.security, com.sap.esb.secu
re.parameter.impl, com.sap.it.nm.concurrent, com.sap.esb.camel.route.po
licy, com.sap.it.iflow.saxonee, net.sf.saxon.lib, org.springframework.t
ransaction.support, org.springframework.jdbc.datasource, com.sap.esb.da
tastore.wrapper, com.sap.esb.monitoring.cxf.response.log, com.sap.esb.m
onitoring.cxf.runtime.feature, com.sap.it.op.agent.trace.cxf, com.sap.e
sb.size.limiter, com.sap.esb.camel.webservice.endpoint.configurer, com.
sap.esb.webservice.security.crypto.api, com.sap.esb.webservice.policy.a
lternative.selector, org.apache.cxf.ws.security.wss4j, org.apache.wss4j
.common.crypto, javax.security.auth.callback, com.sap.it.iflow.model.ru
ntime, com.sap.it.iflow.model, com.sap.sod.utils.idoc.soap.interceptors
, com.sap.sod.utils.idoc.soap.processors, com.sap.sod.utils.idoc.soap,
com.sap.sod.utils.soap.processors, com.sap.sod.utils.soap, com.sap.sod.
utils.encoding.processors, com.sap.sod.utils.cxf.interceptors, com.sap.
it.rt.scc.proxy, com.sap.esb.camel.jdbc.inprogress.repository, com.sap.
esb.camel.jdbc.idempotency.repository, com.sap.esb.camel.webservice.cxf
binding, com.sap.esb.camel.mpl.access, com.sap.esb.camel.eip.splitter,
com.sap.esb.camel.xmljson, com.sap.it.op.ed, com.sap.it.op.agent.ed.plu
gins.camel.api, com.sap.esb.webservice.authorization.supplier.api, org.
apache.cxf.interceptor.security, org.apache.camel.language.xpath, javax
.xml.transform.sax, net.sf.saxon.xpath, net.sf.saxon,com.sap.esb.applic
ation.services.cxf.interceptor,com.sap.esb.security,com.sap.it.op.agent
.api,com.sap.it.op.agent.collector.camel,com.sap.it.op.agent.collector.
cxf,com.sap.it.op.agent.mpl,javax.jms,javax.jws,javax.wsdl,javax.xml.bi
nd.annotation,javax.xml.namespace,javax.xml.ws,org.apache.camel,org.apa
che.camel.builder,org.apache.camel.component.cxf,org.apache.camel.model
,org.apache.camel.processor,org.apache.camel.processor.aggregate,org.ap
ache.camel.spring.spi,org.apache.commons.logging,org.apache.cxf.binding
,org.apache.cxf.binding.soap,org.apache.cxf.binding.soap.spring,org.apa
che.cxf.bus,org.apache.cxf.bus.resource,org.apache.cxf.bus.spring,org.a
pache.cxf.buslifecycle,org.apache.cxf.catalog,org.apache.cxf.configurat
ion.jsse,org.apache.cxf.configuration.spring,org.apache.cxf.endpoint,or
g.apache.cxf.headers,org.apache.cxf.interceptor,org.apache.cxf.manageme
nt.counters,org.apache.cxf.message,org.apache.cxf.phase,org.apache.cxf.
resource,org.apache.cxf.service.factory,org.apache.cxf.service.model,or
g.apache.cxf.transport,org.apache.cxf.transport.common.gzip,org.apache.
cxf.transport.http,org.apache.cxf.transport.http.policy,org.apache.cxf.
workqueue,org.apache.cxf.ws.rm.persistence,org.apache.cxf.wsdl11,org.os
gi.framework,org.slf4j,org.springframework.beans.factory.config,com.sap
.esb.camel.security.cms,org.apache.camel.spi,com.sap.esb.webservice.aud
it.log,com.sap.esb.camel.endpoint.configurator.api,com.sap.esb.camel.jd
bc.idempotency.reorg,javax.sql,org.apache.camel.processor.idempotent.jd
bc,org.osgi.service.blueprint
Origin-Bundle-Name: Basics_Exception_Subprocess
Origin-Bundle-SymbolicName: Basics_Exception_Subprocess
WorkspaceProfile: iflmap
SAP-ArtifactId: 6687c2b9-3f75-438b-bc23-2fb08a9c546f
Import-Service: com.sap.esb.camel.security.cms.SignatureSplitter;multip
le:=false, org.apache.cxf.ws.rm.persistence.RMStore;multiple:=false, ja
vax.sql.DataSource;multiple:=false;filter=”(dataSourceName=default)”, j
avax.sql.DataSource;multiple:=false;filter=”(name=wrapper)”, com.sap.es
b.security.TrustManagerFactory;multiple:=false, com.sap.esb.security.Ke
yManagerFactory;multiple:=false, com.sap.esb.camel.endpoint.configurato
r.api.EndpointConfigurator;multiple:=false;filter=”(endpointType=SFTP)”
,com.sap.esb.webservice.audit.log.AuditLogger
SAP-PreDeployment:
SAP-StartOrder: 150
Require-Capability: sap-HTTPS;resolution:=optional,SAP-ProcessDirect;res
olution:=optional
Bundle-ClassPath: .
But if we just deploy the blueprint locally, we get this (much less, right?):
And as you can see, karaf is kind enough to highlight the packages we depend on.
So, if we were to resolve it, how would we proceed?
Well, we can look (via our webshell of course) for bundles exporting the packages with another command “package:exports -p XXX“
For example, this is who exports com.sap.it.op.agent.* (command implicitly supports wildcards)
And then once you locate that jar within war folder (“bundle:list -l N” – remember the part1?) you get another dependency tree (yes, recursive…).
And you also need to deal with camel components that are not listed in “bundle:headers N” and also services (for those you would issue remote command “service:list” to get huge list of services bundles provide to each other)
Believe it or not, this is how I was dealing with that before I wrote my simple manifest parser (and this is why we need whole war downloaded locally to resolve it)
BTW this is also the reason we only currently support components I was able to resolve manually (those are listed in setup script), so NOT every scenario will work for now.
Ok, so when we resolve (mostly by commenting out beans in blueprint) dependencies for local blueprint, we have something like that:
What that means is we have manual task of adapting the remote blueprint to a “runnable” version based on components we support locally.
And btw this is how it would look like in karaf if I uncommented something that we don’t have installed locally (for example, that “sap-util” we saw earlier)
That’s lame dude..
Ok, but this moment you can probably think that it is a pain in the ass to deal with all that – and you would be mostly right.
And unfortunately currently I cannot suggest any better options rather than Manuel (btw you can try asking AI to adapt blueprint for you 
), so unless you see value in having some local automated tests, you can disregard all that stuff I wrote or showed here in this series.
Otherwise let’s see some examples we have in our samples.
Again, here we will try to prove that behaviour of local blueprints we made from remote ones is indeed what we would like to test.
And obviously, we are NOT trying to test input/output, monitoring or other stuff like that – we just want to test our data transformation and routing logic.
Important thing to notice here is that Camel is payload-agnostic, meaning we are not bound to certain input/output adapters, and we can replace them with anything else as long as message body we get from those other adapters is the same as we expected. Camel does not care.
You can do the following thought experiment: imagine you have soap endpoint, but you immediately put message in jms queue (typical asynchronous InOnly scenario).
What would be the message body once some route gets it from that queue?
Would it be equivalent to polling the envelope body from ftp?
Samples
In this section we will take a look at some iflows from guidelines and compare outlines of remote blueprints to local ones and decide if we have equivalence in regards to what we would like to test..
It is suggested that you visit the links to guidelines provided in each case to make sure you are familiar with how they were designed.
And of course, we have all the stuff here in our repo
Basics_Attachments_Create
This is the soap sender iflow from guidelines where we create attachments in script steps.
I am going to start with this one to answer my own question I asked earlier regarding the equivalence in thought experiment.
And the answer would be “somewhat” because SOAP envelope not only has body, but headers, so polling the body from ftp we would not be able to get those.
And it becomes even more complex when request has attachments (then it is a multipart document already) so we need to somehow unmarshal it back (I failed to do so so far btw)
But even in our case when we “just create attachments” we also end up having multipart response in the end which I replaced with split.
Therefore this case would NOT be equivalent to me, because outputs would NOT be identical.
Anyway, here’s what we have as an outline (here and afterwards: left remote, right local):
Basics_Sender_Initiated_External
Next up we have this guy – very simple (originally) Sender Initiated scenario
For this (as I told in very first video) I originally replaced odata adapter with pollEnrich as I failed to resolve dependencies for odata adapter.
But then I figured out that we still can have http client (ahc adapter) and decided to implement a way to get the same xml we receive from “real’ odata request but with ahc ($format=json) and some scripting (yes, I know I promised to stay as low-code as possible)
Therefore our iflow (and 3 next) have almost identical local process (another route) to fetch xml via ahc or odata based on the header X-Use-Adapter.
Why would I do this? Well, to somehow be able to obtain the data for pollEnrich.
So, the outlines look like this:
Here we see that we indeed have another camel route Process_20 which we call via direct (because we are within one camelContext, remember?)
We don’t see pollEnrich on the right side because it does not have endpoint uri therefore my simple magic (huge thanks to the guy who implemented recursive filter for ClientTreeBinding that works nice with xml model we use there) that just displays anything having to/from uri=”…”
Also we see that store/restore headers is commented out (because well we don’t care)
And in this case I would say those two ARE equivalent, because for ahc case I get literally the same result, while in odata / pollEnrich case I get the same body.
And in next 3 iflows we are going to do some transformations.
Basics_XML_To_WSDL_Mapping (Message Mapping)
Ok, next we are going to finally do something to our body as described here
Again, we consider our local blueprint equivalent to remote if we get the same output for the same input, BUT also keep as much of original blueprint intact as possible (ideally, we just replace inputs and outputs with adapters we support)
So, lets see what we have in this case:
And again, we may agree that those would be equivalent.
Basics_XSLT_Mapping (XSLT Mapping)
Next up we have this guy also from “Access headers and prioperties” of the guide (looks like its just in the alphabetical order there, but I did not care)
Notice here that our main route is now at the top (idk how “iflow compiler” decides that), but also that paths to xsl stylesheet are different as we had to find a way to somehow reach our xslt file (originally unlike with message mapping which is a separate bundle, it is just a local xsl file somewhere inside our bundle – literally in dir “mapping/product-xslt-mapping.xsl).
But this local blueprint http url is a legal parameter coming from standard component xslt-saxon.
Also in this case we ARE able to access headers from xslt stylesheet indeed unlike if we were to use local xslt tooling – like this one for example – because of course it would complain about unknown setHeader function (I believe coming from those saxonExtensionFunctions
Error XPST0017:
Static error in XPath on line 21 in xslt/product-xslt-mapping.xsl {cpi:setHeader($exchange, ‘context’, ‘ModelingBasics-HeaderPropertiesInXSLT’)}: Unknown function Q{http://sap.com/it/}setHeader()
This of course is another point regarding designing locally testable artifacts.
If we comment it out, then we might run something like that from our project root (having xslt3 installed as global node module):
xslt3 -xsl:ftp/Basics_XSLT_Mapping/xslt/product-xslt-mapping.xsl -s:ftp/Basics_XSLT_Mapping/odata/HT-1112.xml quantity=3 orderId=100500
And get the expected xml even without running camel route at all (again, a rather trivial observation).
Which way of testing you prefer is up to you, but I’d say the closer to “original” tools we are the better.
Anyway, I would say in this case we also DO have equivalence.
Basics_XPATH_And_Conditions (Filter by maxPrice)
This one was supposed to be also a simple scenario, but suddenly I spent more than 4 hours troubleshooting it when it didn’t work for me
It’s the third one from the same chapter.
As you maybe noticed, we don’t use json to xml converter, because it failed to convert json structure that would give us an array of Products
So I ended up writing another small script “printing out xml”… smh..
But the real issue I mentioned earlier was that after the xpath “filter” (which just happens to be a setBody eip) I was getting the empty body…
Extra – JMS cron replicator
In conclusion I want to share the first “real” iflow I am building to replicate data from s/4 to some other system.
The requirements were to handle both single and mass replication scenario, so I decided to use queue.
The logic is rather simple: we start periodically via cron, use local variable to track lastSync, then fetch all objects changed after that timestamp from odata v4 service, and process them (json-to-json mapping).
The main cron iflow looks like this.
Please correct me if I got it wrong, but I understood that in most cases S/4 odatav4 services do NOT support $format=xml, therefore standard odata adapter would not be able to process all the stuff in pages automatically.
And therefore we would need to handle the server side pagination ourselves (@odata.nextLink).
So that’s what we do in the loop using script to extract next link from response and set as a property.
The rest is trivial – we basically put Event Message (containing just the ID of object) into the queue so that when it is processed by Processor it fetches the actual data and proceeds.
Ok, so how does the outline look like? Well, rather complicated, so instead of 3 routes we could expect, we see 5 + some sap-retry magic (I currently cannot tell you anything about) intercepting our from jms.
Another interesting thing to notice is that “Send” becomes enrich eip for some reason..
And from that we are calling our ServiceTask_319 route that puts stuff to jms.
So, the question would be, what is the equivalent local blueprint for this guy?
And my answer is – none – as I am not interested in testing this whole thing locally.
The transformation logic happens in my Processor which is extracted from this iflow, while this guy is almost fully generic (except for the queue name maybe if it cannot be externalized)
And obviously, I have another guy there (to handle single object case) which is also a wrapper for Processor
But as you would expect – this is all it does (basically, just converting received payload to xml body):
And the Processor itself fetching data from S4 and doing json-json mapping would be almost identical to our Message Mapping example, but with addition of Exception Handling of course when dealing with responses of that system we communicate to replicate data to.
And that is what I will “localize” and test (and yes, we can also throw exceptions on camel ourselves with something like throwException).
Maybe out of sheer curiosity I will add some small post to this series about dealing with jms queues in local karaf, cuz it seems to be possible (at least in this karaf tutorial I spotted it, so the issue could be with dependencies)
Conclusion
To conclude this part I will come back to the beginning of my post and reiterate my statement:
When we know what we want to test, and designed our iflows accordingly, we can establish the equivalence between local and remote blueprints in regards to behaviour we intend to test.
And as long as sap does not charge extra for “internal” messages (within cpi perimeter like process direct and queues) I see no reason NOT do properly decompose iflows into smaller parts (often even generic), so that even though in the end we have more artifacts, the are well-designed, maintainable and testable.
This applies even if you don’t want to do this local testing stuff, and only do CPI development.
After all, having a low-code platform does not necessarily mean we must produce poorly designed solutions there.
We can do better.
Going back to our favourite picture from the top of the blog: it costs you something to design and implement your testable code (amber “Code Under Test”), but also to maintain your tests (green “Test Case” guys to the left) AND mocks/stubs for external systems etc (green “Test Double” guys to the right).
And here we do have something similar to this: if our “remote blueprint” is “Code Under Test”, then “local blueprint” becomes something of a blend of of a Test Case + Code Under Test
(so maybe we need more than one local blueprint to be able to “control” our Test Doubles which become idk what in this case… yeah..).
It’s a rather new insight (and potential direction for a tool development) I got while finishing this part, so I will try to cover that in next part (will also consider using aforementioned intercept clause), and maybe make us another picture.
For now I hope we can agree that in certain cases local blueprint will be equivalent to remote one, it’s just that we really don’t want to adapt it manually each time we change the remote one (potentially introducing errors) – we want some kind of “automatic assumed equivalence” each time we press “Deploy to karaf” button in the tool.
Another demo
But for now a not-so-short demo this time for XPath scenario with some local debugging (and odata adapter stuff).
00:00 Intro and remote iflow test
04:39 Local blueprint odata dependencies stuff
20:21 Embrace the failure and move on to xpath issues we also have )
36:46 The last part would be pollEnrich and mock data
Link back to index: https://community.sap.com/t5/technology-blogs-by-members/blog-series-on-my-cpi-camel-learning-journey/ba-p/14031053 Problem statementOk, in our previous demo we finally had to admit that we cannot jut use the blueprint CPI produces.This is rather unfortunate because like I said in the video, it brings us back to this pictureHere we see that Green stuff is what we intend to control through test cases design and mocks/stubs.But the most important part is Amber – our Code Under Test.And of course, we are not allowed to modify our code in order to put it under test.This is a big problem, because what that really means for us – is in all cases I showed earlier (and will continue showing), we were testing locally something different than original iflow.I guess, I was referring to it as “local blueprint” (from blueprint-local.xml) while the original one (from OSGI-INFO/bluerint/beans.xml) – as “remote blueprint”.Did it all make any sense then?Well, maybe – if we can establish the equivalence between them in regards to behaviour we intend to test.But why not simply test remote blueprint?Remember the simple sender initiated scenario – we serve the http endpoint to reply with some stuff we fetch from remote odata service.Not only that means we need to run the http server somehow (and dispatch inbound requests to it in cloud), but we also deal with authentication (remember that jwt “ESBMessaging.send” scope?)And that tightly couples us to BTP and CPI runtime and environment (components they have there to interact with BTP).A little bit of osgi finallySo, how does that work in OSGI world?Well, bundles provide services to each other.And so in its META-INF/MANIFEST.MF bundle can define what it imports, exports and provides.This is essential for us (especially in terms of local testing) because this is also how our script bundles and message mapping bundles are discovered/linked by our iflow bundles (remotely) and blueprints (locally).For example, this is how dependencies look like for our friend Basics_Exception_Subprocess on cpi and locally.Just to show you that it is A LOT, so just running it locally would actually mean having the same “complete cloud environment” on your machine.We use webshell to run “bundle:headers N” command to get this list, but as we have it deployed as bundle, I would expect it to exactly match manifest (to the right)I apologise for how it looks, but code snippets suck so much here that eventually I just put stuff into the table (and at least it doesn’t add those bloody empy lines before and after)…Basics_Exception_Subprocess (947)———————————Manifest-Version = 1.0Origin-Bundle-Name = Basics_Exception_SubprocessOrigin-Bundle-SymbolicName = Basics_Exception_SubprocessSAP-ArtifactId = 7c7d554e-8b2d-4ae7-aeb7-0f1cec7d87f2SAP-BundleType = IntegrationFlowSAP-NodeType = IFLMAPSAP-PreDeployment = SAP-RuntimeProfile = iflmapSAP-StartOrder = 150WorkspaceProfile = iflmap Bundle-ClassPath = .Bundle-ManifestVersion = 2Bundle-Name = Basics_Exception_SubprocessBundle-SymbolicName = Basics_Exception_Subprocess; singleton:=trueBundle-Version = 1.0.0 Import-Service = com.sap.esb.camel.security.cms.SignatureSplitter;multiple:=false,org.apache.cxf.ws.rm.persistence.RMStore;multiple:=false,javax.sql.DataSource;multiple:=false;filter=(dataSourceName=default),javax.sql.DataSource;multiple:=false;filter=(name=wrapper),com.sap.esb.security.TrustManagerFactory;multiple:=false,com.sap.esb.security.KeyManagerFactory;multiple:=false,com.sap.esb.camel.endpoint.configurator.api.EndpointConfigurator;multiple:=false;filter=(endpointType=SFTP),com.sap.esb.webservice.audit.log.AuditLoggerRequire-Capability = sap-HTTPS;resolution:=optional Import-Package = org.apache.camel.processor.errorhandler,com.sap.esb.camel.http.ahc.configurer.retry;version=”[1,4)”,com.sap.esb.camel.http.ahc.configurer;version=”[1,2)”,com.sap.it.nm.security,com.sap.esb.secure.parameter.impl,com.sap.it.nm.concurrent,com.sap.esb.camel.route.policy,com.sap.it.iflow.saxonee,net.sf.saxon.lib,org.springframework.transaction.support,org.springframework.jdbc.datasource,com.sap.esb.datastore.wrapper,com.sap.esb.monitoring.cxf.response.log,com.sap.esb.monitoring.cxf.runtime.feature,com.sap.it.op.agent.trace.cxf,com.sap.esb.size.limiter,com.sap.esb.camel.webservice.endpoint.configurer,com.sap.esb.webservice.security.crypto.api,com.sap.esb.webservice.policy.alternative.selector,org.apache.cxf.ws.security.wss4j,org.apache.wss4j.common.crypto,javax.security.auth.callback,com.sap.it.iflow.model.runtime,com.sap.it.iflow.model,com.sap.sod.utils.idoc.soap.interceptors,com.sap.sod.utils.idoc.soap.processors,com.sap.sod.utils.idoc.soap,com.sap.sod.utils.soap.processors,com.sap.sod.utils.soap,com.sap.sod.utils.encoding.processors,com.sap.sod.utils.cxf.interceptors,com.sap.it.rt.scc.proxy,com.sap.esb.camel.jdbc.inprogress.repository,com.sap.esb.camel.jdbc.idempotency.repository,com.sap.esb.camel.webservice.cxfbinding,com.sap.esb.camel.mpl.access,com.sap.esb.camel.eip.splitter,com.sap.esb.camel.xmljson,com.sap.it.op.ed,com.sap.it.op.agent.ed.plugins.camel.api,com.sap.esb.webservice.authorization.supplier.api,org.apache.cxf.interceptor.security,org.apache.camel.language.xpath,javax.xml.transform.sax,net.sf.saxon.xpath,net.sf.saxon,com.sap.esb.application.services.cxf.interceptor,com.sap.esb.security,com.sap.it.op.agent.api,com.sap.it.op.agent.collector.camel,com.sap.it.op.agent.collector.cxf,com.sap.it.op.agent.mpl,javax.jms,javax.jws,javax.wsdl,javax.xml.bind.annotation,javax.xml.namespace,javax.xml.ws,org.apache.camel,org.apache.camel.builder,org.apache.camel.component.cxf,org.apache.camel.model,org.apache.camel.processor,org.apache.camel.processor.aggregate,org.apache.camel.spring.spi,org.apache.commons.logging,org.apache.cxf.binding,org.apache.cxf.binding.soap,org.apache.cxf.binding.soap.spring,org.apache.cxf.bus,org.apache.cxf.bus.resource,org.apache.cxf.bus.spring,org.apache.cxf.buslifecycle,org.apache.cxf.catalog,org.apache.cxf.configuration.jsse,org.apache.cxf.configuration.spring,org.apache.cxf.endpoint,org.apache.cxf.headers,org.apache.cxf.interceptor,org.apache.cxf.management.counters,org.apache.cxf.message,org.apache.cxf.phase,org.apache.cxf.resource,org.apache.cxf.service.factory,org.apache.cxf.service.model,org.apache.cxf.transport,org.apache.cxf.transport.common.gzip,org.apache.cxf.transport.http,org.apache.cxf.transport.http.policy,org.apache.cxf.workqueueorg.apache.cxf.ws.rm.persistence,org.apache.cxf.wsdl11,org.osgi.framework,org.slf4j,org.springframework.beans.factory.config,com.sap.esb.camel.security.cms,org.apache.camel.spi,com.sap.esb.webservice.audit.log,com.sap.esb.camel.endpoint.configurator.api,com.sap.esb.camel.jdbc.idempotency.reorg,javax.sql,org.apache.camel.processor.idempotent.jdbc,org.osgi.service.blueprintManifest-Version: 1.0Bundle-ManifestVersion: 2Bundle-Name: Basics_Exception_SubprocessBundle-SymbolicName: Basics_Exception_SubprocessBundle-Version: 1.0.0SAP-BundleType: IntegrationFlowSAP-NodeType: IFLMAPSAP-RuntimeProfile: iflmapImport-Package: org.apache.camel.processor.errorhandler, com.sap.esb.camel.http.ahc.configurer.retry;version=”[1,4)”, com.sap.esb.camel.http.ahc.configurer;version=”[1,2)”, com.sap.it.nm.security, com.sap.esb.secure.parameter.impl, com.sap.it.nm.concurrent, com.sap.esb.camel.route.policy, com.sap.it.iflow.saxonee, net.sf.saxon.lib, org.springframework.transaction.support, org.springframework.jdbc.datasource, com.sap.esb.datastore.wrapper, com.sap.esb.monitoring.cxf.response.log, com.sap.esb.monitoring.cxf.runtime.feature, com.sap.it.op.agent.trace.cxf, com.sap.esb.size.limiter, com.sap.esb.camel.webservice.endpoint.configurer, com.sap.esb.webservice.security.crypto.api, com.sap.esb.webservice.policy.alternative.selector, org.apache.cxf.ws.security.wss4j, org.apache.wss4j.common.crypto, javax.security.auth.callback, com.sap.it.iflow.model.runtime, com.sap.it.iflow.model, com.sap.sod.utils.idoc.soap.interceptors, com.sap.sod.utils.idoc.soap.processors, com.sap.sod.utils.idoc.soap,com.sap.sod.utils.soap.processors, com.sap.sod.utils.soap, com.sap.sod.utils.encoding.processors, com.sap.sod.utils.cxf.interceptors, com.sap.it.rt.scc.proxy, com.sap.esb.camel.jdbc.inprogress.repository, com.sap.esb.camel.jdbc.idempotency.repository, com.sap.esb.camel.webservice.cxfbinding, com.sap.esb.camel.mpl.access, com.sap.esb.camel.eip.splitter,com.sap.esb.camel.xmljson, com.sap.it.op.ed, com.sap.it.op.agent.ed.plugins.camel.api, com.sap.esb.webservice.authorization.supplier.api, org.apache.cxf.interceptor.security, org.apache.camel.language.xpath, javax.xml.transform.sax, net.sf.saxon.xpath, net.sf.saxon,com.sap.esb.application.services.cxf.interceptor,com.sap.esb.security,com.sap.it.op.agent.api,com.sap.it.op.agent.collector.camel,com.sap.it.op.agent.collector.cxf,com.sap.it.op.agent.mpl,javax.jms,javax.jws,javax.wsdl,javax.xml.bind.annotation,javax.xml.namespace,javax.xml.ws,org.apache.camel,org.apache.camel.builder,org.apache.camel.component.cxf,org.apache.camel.model,org.apache.camel.processor,org.apache.camel.processor.aggregate,org.apache.camel.spring.spi,org.apache.commons.logging,org.apache.cxf.binding,org.apache.cxf.binding.soap,org.apache.cxf.binding.soap.spring,org.apache.cxf.bus,org.apache.cxf.bus.resource,org.apache.cxf.bus.spring,org.apache.cxf.buslifecycle,org.apache.cxf.catalog,org.apache.cxf.configuration.jsse,org.apache.cxf.configuration.spring,org.apache.cxf.endpoint,org.apache.cxf.headers,org.apache.cxf.interceptor,org.apache.cxf.management.counters,org.apache.cxf.message,org.apache.cxf.phase,org.apache.cxf.resource,org.apache.cxf.service.factory,org.apache.cxf.service.model,org.apache.cxf.transport,org.apache.cxf.transport.common.gzip,org.apache.cxf.transport.http,org.apache.cxf.transport.http.policy,org.apache.cxf.workqueue,org.apache.cxf.ws.rm.persistence,org.apache.cxf.wsdl11,org.osgi.framework,org.slf4j,org.springframework.beans.factory.config,com.sap.esb.camel.security.cms,org.apache.camel.spi,com.sap.esb.webservice.audit.log,com.sap.esb.camel.endpoint.configurator.api,com.sap.esb.camel.jdbc.idempotency.reorg,javax.sql,org.apache.camel.processor.idempotent.jdbc,org.osgi.service.blueprintOrigin-Bundle-Name: Basics_Exception_SubprocessOrigin-Bundle-SymbolicName: Basics_Exception_SubprocessWorkspaceProfile: iflmapSAP-ArtifactId: 6687c2b9-3f75-438b-bc23-2fb08a9c546fImport-Service: com.sap.esb.camel.security.cms.SignatureSplitter;multiple:=false, org.apache.cxf.ws.rm.persistence.RMStore;multiple:=false, javax.sql.DataSource;multiple:=false;filter=”(dataSourceName=default)”, javax.sql.DataSource;multiple:=false;filter=”(name=wrapper)”, com.sap.esb.security.TrustManagerFactory;multiple:=false, com.sap.esb.security.KeyManagerFactory;multiple:=false, com.sap.esb.camel.endpoint.configurator.api.EndpointConfigurator;multiple:=false;filter=”(endpointType=SFTP)”,com.sap.esb.webservice.audit.log.AuditLoggerSAP-PreDeployment:SAP-StartOrder: 150Require-Capability: sap-HTTPS;resolution:=optional,SAP-ProcessDirect;resolution:=optionalBundle-ClassPath: .But if we just deploy the blueprint locally, we get this (much less, right?):And as you can see, karaf is kind enough to highlight the packages we depend on.So, if we were to resolve it, how would we proceed?Well, we can look (via our webshell of course) for bundles exporting the packages with another command “package:exports -p XXX”For example, this is who exports com.sap.it.op.agent.* (command implicitly supports wildcards)And then once you locate that jar within war folder (“bundle:list -l N” – remember the part1?) you get another dependency tree (yes, recursive…).And you also need to deal with camel components that are not listed in “bundle:headers N” and also services (for those you would issue remote command “service:list” to get huge list of services bundles provide to each other)Believe it or not, this is how I was dealing with that before I wrote my simple manifest parser (and this is why we need whole war downloaded locally to resolve it)BTW this is also the reason we only currently support components I was able to resolve manually (those are listed in setup script), so NOT every scenario will work for now.Ok, so when we resolve (mostly by commenting out beans in blueprint) dependencies for local blueprint, we have something like that:What that means is we have manual task of adapting the remote blueprint to a “runnable” version based on components we support locally.And btw this is how it would look like in karaf if I uncommented something that we don’t have installed locally (for example, that “sap-util” we saw earlier)That’s lame dude..Ok, but this moment you can probably think that it is a pain in the ass to deal with all that – and you would be mostly right.And unfortunately currently I cannot suggest any better options rather than Manuel (btw you can try asking AI to adapt blueprint for you ), so unless you see value in having some local automated tests, you can disregard all that stuff I wrote or showed here in this series.Otherwise let’s see some examples we have in our samples.Again, here we will try to prove that behaviour of local blueprints we made from remote ones is indeed what we would like to test.And obviously, we are NOT trying to test input/output, monitoring or other stuff like that – we just want to test our data transformation and routing logic.Important thing to notice here is that Camel is payload-agnostic, meaning we are not bound to certain input/output adapters, and we can replace them with anything else as long as message body we get from those other adapters is the same as we expected. Camel does not care.You can do the following thought experiment: imagine you have soap endpoint, but you immediately put message in jms queue (typical asynchronous InOnly scenario).What would be the message body once some route gets it from that queue?Would it be equivalent to polling the envelope body from ftp?SamplesIn this section we will take a look at some iflows from guidelines and compare outlines of remote blueprints to local ones and decide if we have equivalence in regards to what we would like to test..It is suggested that you visit the links to guidelines provided in each case to make sure you are familiar with how they were designed.And of course, we have all the stuff here in our repoBasics_Attachments_CreateThis is the soap sender iflow from guidelines where we create attachments in script steps.I am going to start with this one to answer my own question I asked earlier regarding the equivalence in thought experiment.And the answer would be “somewhat” because SOAP envelope not only has body, but headers, so polling the body from ftp we would not be able to get those.And it becomes even more complex when request has attachments (then it is a multipart document already) so we need to somehow unmarshal it back (I failed to do so so far btw)But even in our case when we “just create attachments” we also end up having multipart response in the end which I replaced with split.Therefore this case would NOT be equivalent to me, because outputs would NOT be identical.Anyway, here’s what we have as an outline (here and afterwards: left remote, right local):Basics_Sender_Initiated_ExternalNext up we have this guy – very simple (originally) Sender Initiated scenarioFor this (as I told in very first video) I originally replaced odata adapter with pollEnrich as I failed to resolve dependencies for odata adapter.But then I figured out that we still can have http client (ahc adapter) and decided to implement a way to get the same xml we receive from “real’ odata request but with ahc ($format=json) and some scripting (yes, I know I promised to stay as low-code as possible)Therefore our iflow (and 3 next) have almost identical local process (another route) to fetch xml via ahc or odata based on the header X-Use-Adapter.Why would I do this? Well, to somehow be able to obtain the data for pollEnrich.So, the outlines look like this:Here we see that we indeed have another camel route Process_20 which we call via direct (because we are within one camelContext, remember?)We don’t see pollEnrich on the right side because it does not have endpoint uri therefore my simple magic (huge thanks to the guy who implemented recursive filter for ClientTreeBinding that works nice with xml model we use there) that just displays anything having to/from uri=”…”Also we see that store/restore headers is commented out (because well we don’t care)And in this case I would say those two ARE equivalent, because for ahc case I get literally the same result, while in odata / pollEnrich case I get the same body.And in next 3 iflows we are going to do some transformations.Basics_XML_To_WSDL_Mapping (Message Mapping)Ok, next we are going to finally do something to our body as described hereAgain, we consider our local blueprint equivalent to remote if we get the same output for the same input, BUT also keep as much of original blueprint intact as possible (ideally, we just replace inputs and outputs with adapters we support)So, lets see what we have in this case:And again, we may agree that those would be equivalent.Basics_XSLT_Mapping (XSLT Mapping)Next up we have this guy also from “Access headers and prioperties” of the guide (looks like its just in the alphabetical order there, but I did not care)Notice here that our main route is now at the top (idk how “iflow compiler” decides that), but also that paths to xsl stylesheet are different as we had to find a way to somehow reach our xslt file (originally unlike with message mapping which is a separate bundle, it is just a local xsl file somewhere inside our bundle – literally in dir “mapping/product-xslt-mapping.xsl).But this local blueprint http url is a legal parameter coming from standard component xslt-saxon.Also in this case we ARE able to access headers from xslt stylesheet indeed unlike if we were to use local xslt tooling – like this one for example – because of course it would complain about unknown setHeader function (I believe coming from those saxonExtensionFunctionsError XPST0017:Static error in XPath on line 21 in xslt/product-xslt-mapping.xsl {cpi:setHeader($exchange, ‘context’, ‘ModelingBasics-HeaderPropertiesInXSLT’)}: Unknown function Q{http://sap.com/it/}setHeader()This of course is another point regarding designing locally testable artifacts.If we comment it out, then we might run something like that from our project root (having xslt3 installed as global node module):xslt3 -xsl:ftp/Basics_XSLT_Mapping/xslt/product-xslt-mapping.xsl -s:ftp/Basics_XSLT_Mapping/odata/HT-1112.xml quantity=3 orderId=100500And get the expected xml even without running camel route at all (again, a rather trivial observation).Which way of testing you prefer is up to you, but I’d say the closer to “original” tools we are the better. Anyway, I would say in this case we also DO have equivalence.Basics_XPATH_And_Conditions (Filter by maxPrice)This one was supposed to be also a simple scenario, but suddenly I spent more than 4 hours troubleshooting it when it didn’t work for meIt’s the third one from the same chapter.As you maybe noticed, we don’t use json to xml converter, because it failed to convert json structure that would give us an array of Products So I ended up writing another small script “printing out xml”… smh..But the real issue I mentioned earlier was that after the xpath “filter” (which just happens to be a setBody eip) I was getting the empty body…Original one:<camel:xpath documentType=”javax.xml.transform.sax.SAXSource” factoryRef=”saxonEEXpathFactory” preCompile=”false” resultType=”org.w3c.dom.NodeList”> //Products/Product[./Price < $maxprice]</camel:xpath> So it took me quite some time to figure out that the issue was not the xpath expression or dependencies issue, but rather documentType..And only after I randomly went to camel xpath language docs to find that default documentType was org.w3c.dom.Document rather than some javax.xml.transform.sax.SAXSource it finally worked for me:<camel:xpath documentType=”org.w3c.dom.Document” factoryRef=”saxonEEXpathFactory” preCompile=”false” resultType=”org.w3c.dom.NodeList”> //Products/Product[./Price < $maxprice]</camel:xpath>And now THAT was quite unexpected, as on cpi side it was working just fine.This would be an example when I am not so sure about the equivalence because I changed some internal logic of transformation rather than juts input/output adapters.And if someone knows how to make it work without modifications, please let us know in the comments.Extra – JMS cron replicatorIn conclusion I want to share the first “real” iflow I am building to replicate data from s/4 to some other system.The requirements were to handle both single and mass replication scenario, so I decided to use queue.The logic is rather simple: we start periodically via cron, use local variable to track lastSync, then fetch all objects changed after that timestamp from odata v4 service, and process them (json-to-json mapping).The main cron iflow looks like this.Please correct me if I got it wrong, but I understood that in most cases S/4 odatav4 services do NOT support $format=xml, therefore standard odata adapter would not be able to process all the stuff in pages automatically.And therefore we would need to handle the server side pagination ourselves (@odata.nextLink).So that’s what we do in the loop using script to extract next link from response and set as a property.The rest is trivial – we basically put Event Message (containing just the ID of object) into the queue so that when it is processed by Processor it fetches the actual data and proceeds.Ok, so how does the outline look like? Well, rather complicated, so instead of 3 routes we could expect, we see 5 + some sap-retry magic (I currently cannot tell you anything about) intercepting our from jms.Another interesting thing to notice is that “Send” becomes enrich eip for some reason..And from that we are calling our ServiceTask_319 route that puts stuff to jms.So, the question would be, what is the equivalent local blueprint for this guy?And my answer is – none – as I am not interested in testing this whole thing locally.The transformation logic happens in my Processor which is extracted from this iflow, while this guy is almost fully generic (except for the queue name maybe if it cannot be externalized) And obviously, I have another guy there (to handle single object case) which is also a wrapper for ProcessorBut as you would expect – this is all it does (basically, just converting received payload to xml body):And the Processor itself fetching data from S4 and doing json-json mapping would be almost identical to our Message Mapping example, but with addition of Exception Handling of course when dealing with responses of that system we communicate to replicate data to.And that is what I will “localize” and test (and yes, we can also throw exceptions on camel ourselves with something like throwException).Maybe out of sheer curiosity I will add some small post to this series about dealing with jms queues in local karaf, cuz it seems to be possible (at least in this karaf tutorial I spotted it, so the issue could be with dependencies)ConclusionTo conclude this part I will come back to the beginning of my post and reiterate my statement:When we know what we want to test, and designed our iflows accordingly, we can establish the equivalence between local and remote blueprints in regards to behaviour we intend to test.And as long as sap does not charge extra for “internal” messages (within cpi perimeter like process direct and queues) I see no reason NOT do properly decompose iflows into smaller parts (often even generic), so that even though in the end we have more artifacts, the are well-designed, maintainable and testable.This applies even if you don’t want to do this local testing stuff, and only do CPI development.After all, having a low-code platform does not necessarily mean we must produce poorly designed solutions there.We can do better.Going back to our favourite picture from the top of the blog: it costs you something to design and implement your testable code (amber “Code Under Test”), but also to maintain your tests (green “Test Case” guys to the left) AND mocks/stubs for external systems etc (green “Test Double” guys to the right).And here we do have something similar to this: if our “remote blueprint” is “Code Under Test”, then “local blueprint” becomes something of a blend of of a Test Case + Code Under Test(so maybe we need more than one local blueprint to be able to “control” our Test Doubles which become idk what in this case… yeah..).It’s a rather new insight (and potential direction for a tool development) I got while finishing this part, so I will try to cover that in next part (will also consider using aforementioned intercept clause), and maybe make us another picture.For now I hope we can agree that in certain cases local blueprint will be equivalent to remote one, it’s just that we really don’t want to adapt it manually each time we change the remote one (potentially introducing errors) – we want some kind of “automatic assumed equivalence” each time we press “Deploy to karaf” button in the tool.Another demoBut for now a not-so-short demo this time for XPath scenario with some local debugging (and odata adapter stuff).00:00 Intro and remote iflow test04:39 Local blueprint odata dependencies stuff20:21 Embrace the failure and move on to xpath issues we also have )36:46 The last part would be pollEnrich and mock data Read More Technology Blogs by Members articles
#SAP
#SAPTechnologyblog
