Wednesday, September 19, 2007

SOAP Client for Amazon ECS with XFire

SOAP based Webservices are fairly ubiquitous nowadays, but so far, I had never had a need to build one. I have built several non-SOAP Webservices in the past, but they were all for internal use, so I used various light-weight remoting technologies such as Caucho's Burlap and Hessian, Spring's HttpInvoker, RMI and so on. All of these involve making the API JARs for the service available to the client somehow. SOAP is more like CORBA, in the sense that its WSDL file is similar to the CORBA IDL, and serves the same purpose. Given a WSDL, the client should be able to generate an API locally.

One of the nice things about being a developer is that, when given a tool or technology you are unfamiliar with, you can build in some time into your project to learn it. As a manager, that luxury is denied to me. I have seen managers at past jobs deal with this by doing the unfamiliar work themselves, and handing off the rest to their engineers, but having been on the recieving end of this transaction, I did not want to perpetuate it. Besides, I was working on another equally high-priority project at the time and did not have the bandwidth to commit to a delivery date for this one. So I ended up assigning this work to one of our engineers, which neither he nor I had done before.

Nevertheless, it made me uncomfortable that I did not know enough about what I was asking someone else to do. Besides, given the ubiquity of SOAP, being able to build a SOAP client should be part of the average Java developer's skillset. So I decided to do a small proof-of-concept to learn about how to build a SOAP Webservice client. The service I chose to hit was Amazon's E-Commerce System (Amazon ECS or AWS), a very comprehensive and well-built API that exposes almost every bit of information you can find on their website. You can see their WSDL file here.

For a toolkit, I first chose Axis2, but its wsdl2java tool fails with unsatisfied dependencies when I request adf or jibx data bindings, and hangs when I request XmlBeans bindings. For those unfamiliar with the term "data bindings" (as I was when I started last weekend), its just the generated Java beans representing the types defined in the WSDL file, and the parsing code to convert between the XML and Java. Colleagues have reported success generating APIs from WSDL files using IDEs such as Netbeans and IDEA, and I am sure Eclipse can do it too, but wsdl2java from Axis2 did not work for me on the command line.

I then chose Apache CXF, formerly known as XFire. XFire seems to be the more well-known name, which is why I used it in the title - in the rest of the post I will call it Apache-CXF. I had heard about Apache-CXF when checking out Spring remoting strategies. It is supposed to be lighter weight than Axis2, but I don't know enough about either project to elaborate on what that means. Anyway, its wsdl2java worked great, generating JAXB bindings for me by default. Here is the command I used to generate the API from the AWS WSDL file:

1
2
3
4
5
sujit@sirocco:~/tmp/apache-cxf-2.0.1-incubator/bin$ ./wsdl2java \
  -p net.sujit.amazon.generated \
  -client \
  -d /home/sujit/src/wsclient/src/main/java \
  http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl

This generated a bunch of files in the net/sujit/amazon/generated subdirectory of my Maven2 application. At this point, I was now ready to build my client. Amazon is my (and probably the world's) favorite bookstore, so I decided to build a client that would search for and return a list of books based on a search string. To keep my client separate from the generated code, I put it in a parallel package to the generated code. The client code follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
package net.sujit.amazon.client;

import java.util.ArrayList;
import java.util.List;

import net.sujit.amazon.generated.AWSECommerceService;
import net.sujit.amazon.generated.AWSECommerceServicePortType;
import net.sujit.amazon.generated.Item;
import net.sujit.amazon.generated.ItemAttributes;
import net.sujit.amazon.generated.ItemSearch;
import net.sujit.amazon.generated.ItemSearchRequest;
import net.sujit.amazon.generated.ItemSearchResponse;
import net.sujit.amazon.generated.Items;

import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;

public class MyWebServicesClient {

  private static final Logger LOGGER = Logger.getLogger(MyWebServicesClient.class);
  private static final String ACCESS_KEY = "MySuperSecretHexKey";
  private static final String BOOK_SEARCHINDEX = "Books";
  
  private AWSECommerceServicePortType client;
  
  public void init() throws Exception {
    AWSECommerceService service = new AWSECommerceService();
    this.client = service.getAWSECommerceServicePort();
  }
  
  public List<MyBook> getSearchResults(String keywords) {
    List<MyBook> myBooks = new ArrayList<MyBook>();
    ItemSearch itemSearch = new ItemSearch();
    itemSearch.setAWSAccessKeyId(ACCESS_KEY);
    ItemSearchRequest request = new ItemSearchRequest();
    request.setKeywords(keywords);
    request.setCondition("All");
    request.setSearchIndex(BOOK_SEARCHINDEX);
    request.getResponseGroup().add("ItemAttributes");
    itemSearch.getRequest().add(request);
    ItemSearchResponse response = client.itemSearch(itemSearch);
    List<Items> itemsList = response.getItems();
    for (Items items : itemsList) {
      List<Item> itemList = items.getItem();
      for (Item item : itemList) {
        MyBook myBook = new MyBook();
        myBook.setAsin(item.getASIN());
        myBook.setUrl(item.getDetailPageURL());
        ItemAttributes attributes = item.getItemAttributes();
        myBook.setTitle(attributes.getTitle());
        myBook.setAuthor(StringUtils.join(attributes.getAuthor().iterator(), ", "));
        myBook.setPublisher(attributes.getPublisher());
        myBook.setPublicationDate(attributes.getPublicationDate());
        myBooks.add(myBook);
      }
    }
    return myBooks;
  }
}

As you can see, my application client instantiates the service using the service stub and gets a reference to the underlying client proxy. This is done in the init() method. Once you have that, you are golden, and the rest of the code is just application code calling the remote methods via the proxy.

The getSearchResults() method represents the actual application code. The method takes a search string as its argument. It then instantiates an ItemSearch object with the AWS Access key, then builds an ItemSearchRequest with the search string, and assigns the ItemSearchRequest to the ItemSearch object. Executing the ItemSearch object's execute() method yields a ItemSearchResponse, which I then query to pull out the information from the web service into a List of MyBook view beans. The MyBook bean is a simple data holder, with the fields as shown below. The getters and setters have been omitted in the interest of space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
package net.sujit.amazon.client;

public class MyBook {
  private String asin;
  private String title;
  private String url;
  private String author;
  private String publisher;
  private String publicationDate;
  ...  
}

More information on the various methods available in AWS and the parameters that they take are available in the Amazon ECS Web Developer's guide. As mentioned before, the service is very comprehensive, so its advisable to go through the guide if you want to do anything serious with it.

I use a JUnit test to actually call this method. This could also have been done using a main() method on the MyWebServicesClient.java file. The JUnit test looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
package net.sujit.amazon.client;

import java.util.List;

import org.apache.log4j.Logger;
import org.junit.Before;
import org.junit.Test;

public class MyWebServicesClientTest {

  private static final Logger LOGGER = Logger.getLogger(MyWebServicesClientTest.class);
  
  private MyWebServicesClient client;
  
  @Before
  public void setUp() throws Exception {
    client = new MyWebServicesClient();
    client.init();
  }
  
  @Test
  public void testGetSearchResults() throws Exception {
    List<MyBook> books = client.getSearchResults("java web services");
    LOGGER.debug("#-books:" + books.size());
    for (MyBook book : books) {
      LOGGER.debug("--");
      LOGGER.debug(book.toString());
    }
  }
}

The results for a search of "java web services" (see test above) return the first 10 results of the search. I did not actually build an UI for it, so I am not going to include the results here, but building an UI should be a very simple task.

As you can see, the actual code you have to write to build a SOAP client is minimal. In that sense, I can see why SOAP is so popular as an external Webservice framework, even though the XML itself is so horribly bloated compared to other XML based remoting protocols. Building a client should take very little time, if you know the things to do before you start. I outline the list of steps I had to do to get my SOAP client up and working.

  1. Download apache-cxf from the project website.
  2. Sign up for a free Webservices account with Amazon to get my access key.
  3. Generate a Maven2 Java application
  4. Add to my default pom.xml the JAR files listed in the "all CXF usage" section of the lib/WHICH_JARS file of the apache-cxf distribution. They missed the wsdl4j.jar file in the list which I added in later.
  5. Locally install JARs specified in above list but not already available in my repository using mvn install:install-file.
  6. Run the wsdl2java command so that the java files are generated in the right spot in my application.
  7. Develop the MyWebservicesClient.java file to define the services I should call for my use case.
  8. In addition, I defined a view bean (MyBook.java) to be able to collect the information out of the service into my application.
  9. Write a JUnit test to run the client.

I hope this post was useful. I did find some articles on the Internet about how to develop a SOAP web client, but most of these are from companies hawking their IDE, IDE plugins or other visual products, and you have to read between the lines to see how everything fits together. My goal of building the proof of concept described here was to understand how the whole thing works, something I do not get when someone points and clicks his way through an IDE or other visual tool.

9 comments (moderated to prevent spam):

Hemant Patel said...

Hi Sujit,
No words to speak, I wish we may work on some high-end technologies this way :-).

Sujit Pal said...

Thanks for your kind words, Hemant.

Anonymous said...

I like your article above and I also like CXF more than Axis2. But I do not think CXF's documents are as good as those of Axis2. Really hope it could be improved soon.

Sujit Pal said...

Thanks John, glad you liked the article. As a newbie to web services clients, what I liked about CXF was the ease with which I could get up and running. However, as I work more with it, I see that certain things are hard to do, or maybe I just don't know enough. What I am after is to be able to set the client socket timeout parameter dynamically, so far I haven't been able to do it, maybe its time to look at the sources now :-).

Sujit Pal said...

Here is a hack I came up with to make the client wait forever until the server comes back with a response. Its not as cool as being able to set it dynamically, but this should work just as well:
Client clientProxy = ClientProxy.getClient(client);
HTTPConduit conduit = (HTTPConduit) clientProxy.getConduit();
HTTPClientPolicy policy = new HTTPClientPolicy();
policy.setConnectionTimeout(0L);
policy.setReceiveTimeout(0L);
conduit.setClient(policy);

Just thought I'd share...

Mohit Agrawal said...

If I have to use this client from within any web application running on Tomcat, can i do it directly without any extra configuration settings?

Sujit Pal said...

Hi Mohit, AFAIK, there should be no extra configuration that is needed for it to work within a web application.

Unknown said...

Hi,

I am new to CXF and tried to run this example...

I followed the steps and registeres at AWS and got a access key

When I try to run the junit from Eclipse this is what I get.

When I debugged...i found client.getSearchResults("java web services"); does not return anything.

what is incorrect?


log4j:WARN No appenders could be found for logger (org.apache.cxf.bus.spring.BusApplicationContext).
log4j:WARN Please initialize the log4j system properly.
Nov 5, 2008 1:41:28 PM org.apache.cxf.bus.spring.BusApplicationContext getConfigResources
INFO: No cxf.xml configuration file detected, relying on defaults.

Sujit Pal said...

Hi Jagadeesh, you may want to check whether you are opted-in to the ECS service that you are trying to query. You would need to go into the Amazon site, select the service, and then click the button opting in to use the service. I had this same problem recently when I was trying to access ec2 from the command line.