Anyone Using "Biomart + Java Web Services" ?

Question

[Pierre Lindenbaum]

Hi all, Thanks to Neil, I've discovered that Biomart is not just a HTML front-end but it can also be invoked as a web service.

So, I played with java and the following webservice: http://www.ensembl.org/biomart/martwsdl .

${JAVA_HOME}/bin/wsimport http://www.ensembl.org/biomart/martwsdl

was called to generate the java sources from the WSDL file and I also wrote the following client:

import org.ensembl._80.martservicesoap.*;

public class BioMartClient
 {
 public static void  main(String args[]) throws Exception
  {
  BioMartSoapService service=new BioMartSoapService();
  MartServiceSoap martsoap= service.getBioMartSoapPort();
  for(Mart mart: martsoap.getRegistry())
   {
   System.out.println(mart.getName());
   }
  }
 }

but the program throwed the following exception when it invokes getRegistry():

Exception in thread "main" com.sun.xml.internal.ws.server.UnsupportedMediaException: Unsupported Content-Type: text/html; charset=utf-8 Supported ones are: [text/xml]
        at com.sun.xml.internal.ws.encoding.StreamSOAPCodec.decode(StreamSOAPCodec.java:284)
(...)
        at $Proxy30.getRegistry(Unknown Source)
        at BioMartClient.main(BioMartClient.java:9)

I also tested a few more BioMart servers: Sometimes getRegistry just returns an empty list (http://www.biomart.org/biomart/martwsdl ), sometimes the path martwsdl does not exist ( http://www.wormbase.org/biomart/martwsdl ), etc...

In the end I didn't find any server containing some data that could be processed via the SOAP protocol.

Is it a known issue ? Am I missing something ? Is there a functional BioMart+SOAP server anywhere ?

Thanks,

Update: I've tested it again today (16 jan 2012) http://www.biomart.org/biomart/martwsdl : there was no error but the response for getRegistry is empty.

[Michael]

Actually, I see we are using different marts, could you try the 'real' mart at http://www.biomart.org/biomart/martwsdl ?

[Michael]

Just sent my message to mart-dev, let's see what comes out of it.

[Michael]

I wanted to use a telephone, but was told to use a walkie-talkie instead... For now, no chance for the SOAP service, have to try the REST service instead. The developers say, they are working on porting BioMart to Java, which I appreciate.

[Pierre Lindenbaum]

I saw your messages on the mart-dev mailing Michael. Thanks for that. Yes, some answers were weird "your java thing cannot understand our service ? solution is: use REST..." :-/

Answer 1

[Michael]

Edit: Conclusion: It does not work and is unlikely to work without changing the service.

In the meantime, I would like to ask everyone who has similar experiences with non-functional SOAP/REST web-services, non-parsable WSDLS, non-validating XML-messages or other problems with service interoperability not to give up, but to report to the providers, to challenge them to improve the interoperability!

Those experiencing similar problems with BioMart can for example report to: [e-mail]

Testing bioinformatics web-services can sometimes be a frustrating experience. It looks as if all the specs of interoperability are not worth the (white-) papers they are printed on. I actually have had very little success in connecting to perl, SOAP-lite based web-services like BioMart, Kegg, etc.

Actually, I tried to use a generated Axis2 client and got an exception. Then, I tried to validate the response message in SoapUI from BioMart, and it does not validate!

line 4: Expected element 'mart' instead of 'mart@http://www.biomart.org:80/MartServiceSoap' here in element getRegistryResponse@http://www.biomart.org:80/MartServiceSoap

So it's very unlikely that this is going to work at all, except with perl and SOAP::Lite. There are too many tiny glitches involved. To clean up this mess I intend to contact the Bimart developers and ask them to change their interface. In fact, it seems that nobody has tested that yet with Java, otherwise somebody would have noticed that it does not work.

Maybe, you wish to join me in this effort?

---

Hi Pierre,

actually it would surprise me if there were no problems. This reminds me of some experiences I encountered when I tried to use the highly appraised standardized SOAP web-services world (actually I wanted to ask a question about user's experiences with bioinfo services but didn't get that far). So, it is a bit like the famous radio yerevan, "In principle it's standardized, but everyone has different standards."

However, and not to just bash the providers, I can remember that we had successfully accessed biomart SOAP web-services with SoapUI. That tool is great for testing and building messages. Alternatively try a different SOAP stack like Axis2 to generate the binding.

I might give it a try and generate a simple Axis2 client, then I will edit and provide the java code. So long there could be something wrong with the way you are trying to use the library. Also, Biomart is sensitive to the exact workflow of service calls.

This is what I just tried in soapui: with SoapUI load the wsdl at: http://www.biomart.org/biomart/martwsdl With other tools (eg curl, see comments), send the xml-message below to the service endpoint address: http://www.biomart.org/biomart/martsoap

then call "getRegistry" as you did with the following message:

<soapenv:Envelope xmlns:soapenv="&lt;a href=" http:="" schemas.xmlsoap.org="" soap="" envelope="" "="" rel="nofollow">http://schemas.xmlsoap.org/soap/envelope/" xmlns:mar="http://www.biomart.org:80/MartServiceSoap">

   <soapenv:Header/>

   <soapenv:Body>

      <mar:getRegistry/>
   </soapenv:Body>
</soapenv:Envelope>

Then you will get this response, which looks very much ok.

<soap:Envelope soap:encodingStyle="&lt;a href=" http:="" schemas.xmlsoap.org="" soap="" encoding="" "="" rel="nofollow">http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <getRegistryResponse xmlns="&lt;a href=" http:="" www.biomart.org:80="" MartServiceSoap"="" rel="nofollow">http://www.biomart.org:80/MartServiceSoap">
         <mart>
            <name xsi:type="xsd:string">ensembl</name>
            <displayName xsi:type="xsd:string">ENSEMBL GENES 57 (SANGER UK)</displayName>
            <database xsi:type="xsd:string">ensembl_mart_57</database>
            <host xsi:type="xsd:string">www.biomart.org</host>
            /biomart/martservice
            <port xsi:type="xsd:string">80</port>
            <visible xsi:type="xsd:int">1</visible>
            <default xsi:type="xsd:int">1</default>
            <serverVirtualSchema xsi:type="xsd:string">default</serverVirtualSchema>
            <includeDatasets xsi:type="xsd:string"/>
            <martUser xsi:type="xsd:string"/>
            <redirect xsi:nil="true" xsi:type="xsd:int"/>
         </mart>

[Pierre Lindenbaum]

Many thanks Michael, I will try with another tool. wsimport works fine with the services created with ${JAVA_HOME}/bin/wsgen. For example I tested it with EBI/Intact

[Pierre Lindenbaum]

Michael, a small typo; your URL should be http://www.biomart.org/biomart/martsoap to martwsdl

[Pierre Lindenbaum]

Hum, I tried CXF/wsdl2java : same (empty) result

[Pierre Lindenbaum]

Michael, a small typo; your URL should be biomart.org/biomart/martsoap , not martwsdl

[Pierre Lindenbaum]

but, as you said

curl --header 'Content-Type: text/xml' --data '[?]
[?]
   [?]
      [?]
   [?]
[?]
' http://www.biomart.org/biomart/martsoap

is OK :-/

[Michael]

correct: biomart.org/biomart/martsoap is the service endpoint, while /martwsdl is the wsdl location. Whith soapui, you just put in the wsdl and the rest is automatic. Good luck!

[Pierre Lindenbaum]

i asked for help on SO : http://stackoverflow.com/questions/2479069

[Pierre Lindenbaum]

I got the answer from the mart mailing list ... hum....

[Michael]

I saw it, looks as if they didn't get what the problem is. I will write to the mart_dev with an in-depth description of the problems in the service and wsdl. That might need a little more time.

Answer 2

[Michael]

Hi again, I was finally able to edit the received message such that it can be validated. The biomart reponse semed to screw up namespace definitions and they use XML-schema instance. :o) The schema instance definitions had no obvious use and did not validate. So I just removed them. If you look at that w3c site you see that W3C says this should never be used. So compare the valid edited reponse massage with the real (but shortened) output. The next step is to make the maintainers have their service send valid XML Here is the edited and schema-valid response, only the first mart given:

<soap:Envelope soap:encodingStyle="&lt;a href=" http:="" schemas.xmlsoap.org="" soap="" encoding="" "="" rel="nofollow">http://schemas.xmlsoap.org/soap/encoding/"

xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:tns="http://www.biomart.org:80/MartServiceSoap"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <tns:getRegistryResponse>
         <mart>
            <name>ensembl</name>
            <displayName>ENSEMBL GENES 57 (SANGER UK)</displayName>
            <database>ensembl_mart_57</database>
            <host>www.biomart.org</host>
            /biomart/martservice
            <port>80</port>
            <visible>1</visible>
            <default>1</default>
            <serverVirtualSchema>default</serverVirtualSchema>
            <includeDatasets/>
            <martUser/>
            <redirect>1</redirect>
         </mart>

      </tns:getRegistryResponse>
   </soap:Body>
</soap:Envelope>

And compare this with the original response:

<soap:Envelope soap:encodingStyle="&lt;a href=" http:="" schemas.xmlsoap.org="" soap="" encoding="" "="" rel="nofollow">http://schemas.xmlsoap.org/soap/encoding/"     
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" 
xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <getRegistryResponse xmlns="&lt;a href=" http:="" www.biomart.org:80="" MartServiceSoap"="" rel="nofollow">http://www.biomart.org:80/MartServiceSoap">
         <mart>
            <name xsi:type="xsd:string">ensembl</name>
            <displayName xsi:type="xsd:string">ENSEMBL GENES 57 (SANGER UK)</displayName>
            <database xsi:type="xsd:string">ensembl_mart_57</database>
            <host xsi:type="xsd:string">www.biomart.org</host>
            /biomart/martservice
            <port xsi:type="xsd:string">80</port>
            <visible xsi:type="xsd:int">1</visible>
            <default xsi:type="xsd:int">1</default>
            <serverVirtualSchema xsi:type="xsd:string">default</serverVirtualSchema>
            <includeDatasets xsi:type="xsd:string"/>
            <martUser xsi:type="xsd:string"/>
            <redirect xsi:nil="true" xsi:type="xsd:int"/>
         </mart>
      </getRegistryResponse>
   </soap:Body>
</soap:Envelope>

[Pierre Lindenbaum]

Thanks Michael !

Answer 3

[Manu Prestat]

The R/BioConductor library biomaRt is also pretty useful for R lovers.

Answer 4

[Gorka]

Hi all,

I have problems too using the WSDL at http://www.biomart.org/biomart/martwsdl. In my case I get an empty List<ResultsRow> in the code after MartServiceSoap.query(), despite the results are actually returned by the server as I have confirmed using Wireshark.

Finally, I've been able to use the service using the WSDL at http://central.biomart.org/martsoap?wsdl. With this WSDL there is a "raw" PortalServiceImpl.getResults() method that accepts the query as XML and returns all the results as a string.

Here there is the automatically generated code for consuming the WSDL:
https://code.google.com/p/s-chpp/source/browse/trunk/java/src/main/java/#java%2Forg%2Fbiomart%2Fapi%2Fsoap

And here how I have managed to get the results:
https://code.google.com/p/s-chpp/source/browse/trunk/java/src/main/java/org/schpp/db/Ensembl.java

I hope that this service support is continued since it is very useful!

Answer 5

[Steve Moss]

The MartService is pretty cool, I discovered it while doing research for my placement at the EBI (see below).

I am/have been working on an EnsEMBL RESTful web service with the EnsEMBL software development team. See these posts from my blog:

• EnsEMBL Geek for a Week part one...
• EnsEMBL Geek for a Week part two (an EnsEMBL RESTful Web Service)...

I implemented it using Perl, Catalyst and JSON, which I think is the way to go?

There is currently no time-frame for completion/public release, but I'll keep things updated on my blog and on here as much as I can. It is something that I am doing around my PhD, and as such my time is limited.

Any recommendations, or ideas on the best way to provide such as service should be appreciated. The prototype seemed to work well, but I need to look at security, caching and other performance issues etc.

[Pierre Lindenbaum]

thanks, but I was more interested by the WSDL/SOAP API (which doesn't work)

[Steve Moss]

Cool, no problem!