Gil's profile at Stack Overflow Coding challenges at Project Euler



Retrofit 2: Code walkthrough

Retrofit is an open source library by Square that turns annotated java interfaces into REST clients. The library is currently in its second beta and changed quite a bit from its first version to better support the Ok stack. In this post we will go through its code, explaining its main techniques and inner workings. This post assumes some knowledge of the library as a user of it.

Retrofit is a simple and small library. In principle you don't need it, you could do all the work directly in okhttp: build requests manually with okhttp3.Builder, then use some custom converter to turn ResponseBodys to your own pojos, and perhaps move callbacks to specific threads with Executor. Yet Retrofit does all this with a simple, easy to understand, annotation-based API.

The first thing to happen when a method is called is its parsing into an internal model, its MethodHandler, that is then cached to avoid future overhead. Afterwards, and on every method call, the method arguments are passed to its handler which has the responsability to create a Call object, a description of a request that can be triggered later either synchronously or asynchronously. Before going into the detail of how requests are built, it is interesting to study a bit how Retrofit links interfaces and method handlers.

Dynamic proxies

A proxy is an object that receives calls and delegate them to another object, perhaps adding custom behavior like security checks or instrumentation. This is a classic software design pattern that can be implemented without any special API. The difference here is the "dynamic" part, that is, the possibility to create on-the-fly implementations out of class tokens that represent service interfaces (class tokens are plain Class<?> objects).

Think about how Retrofit could implement all its service interfaces: it cannot know them in advance, so the regular way to use implements in code is out of question. Nevertheless, logically speaking, to implement an interface you only need to know about the interface itself and the behavior you want to associate with it. Dynamic proxies offer a simple API for it: you pass in an interface by a Class<?> object, you pass in the behavior as an InvocationHandler, and the API creates the proxy instance for you.

Let's see how this is implemented in code. We use the code from Square's repo at commit 90729eb2:

public <T> T create(final Class<T> service) {
  Utils.validateServiceInterface(service);
  // ... more validation here ...      
  return (T) Proxy.newProxyInstance(service.getClassLoader(), 
      new Class<?>[] { service },
      new InvocationHandler() {
        private final Platform platform = Platform.get();

        @Override public Object invoke(Object proxy, Method method, 
            Object... args) throws Throwable {
          // If the method is a method from Object then defer to normal
          // invocation.
          if (method.getDeclaringClass() == Object.class) {
            return method.invoke(this, args);
          }
          if (platform.isDefaultMethod(method)) {
            return platform.invokeDefaultMethod(method, service, proxy, args);
          }
          return loadMethodHandler(method).invoke(args);
        }
      });
}

The key point is the call to Proxy.newProxyInstance(), the one that creates the dynamic proxy. Keep also in mind that a class token does not have to represent an interface but interfaces are required for dynamic proxies, hence this is checked beforehand by the utility method Util.validateServiceInterface(). Take a look to heterogeneous containers in Effective Java, Item 29 to see some other usages of class tokens.

The real logic is in the InvocationHandler, implemented as an anonymous inner class here, which is responsible to load a MethodHandler (and cache it) and then to delegate to its invoke() method, passing over the arguments.

Some additional details: using the interface's class loader makes the proxy and the interface be loaded by the same class loader. This is actually required, because if the class loader cannot see the interface, the JVM will throw an IllegalArgumentException: this is trivial if there is only one interface like here, but could be a problem if the proxy has to implement multiple interfaces loaded by different class loaders.

Finally the platform object defines whether Retrofit is running on Java 8, Android, or something else. In Java 8 we can have default methods on interfaces, so there is a check for that. One further check is to let the dynamic proxy handle methods from Object, for example equals(), hashCode(), etc.

The main idea: annotations parsing

In this section we discuss how Retrofit parses annotations to create a RequestFactory that is then used inside OkHttpCall to create Requests. After this, there will be still some plumbing to do, which is the topic of the next section. Let's take a look how to create a MethodHandler, where the parsing starts:

static MethodHandler create(Retrofit retrofit, Method method) {
  CallAdapter<?> callAdapter = createCallAdapter(method, retrofit);
  Type responseType = callAdapter.responseType();
  Converter<ResponseBody, ?> responseConverter =
      createResponseConverter(method, retrofit, responseType);
  RequestFactory requestFactory = RequestFactoryParser.parse(method,
     responseType, retrofit);
  return new MethodHandler(retrofit.callFactory(), requestFactory, callAdapter,
      responseConverter);
}

A MethodHandler needs four dependencies:

  1. A request factory
  2. A raw call factory for okhttp3.Calls
  3. A call adapter
  4. A response converter

RequestFactory is a simple class that is responsible to create a new Request each time its single method create() is invoked. (Although this is not necessary as a new instance of OkHttpCall is created each time, we will see this later.) Calls, call adapters and respose converters will be discussed in the next section.

When create() is invoked, it does three things:

  1. Create a RequestBuilder: this is the final builder of a Request, it contains various lower level okhttp builders.
  2. Apply an array of RequestActions to the it : those are pending method calls on the builder, created during parsing following the command pattern.
  3. Call build() on the builder and return the Request.

The builder itself is not very complicated, it is responsible for:

The only thing left to see is how RequestActions are created during parsing. Continuing from the code above we call RequestFactoryParse.parse():

static RequestFactory parse(Method method, Type responseType, Retrofit retrofit) {
  RequestFactoryParser parser = new RequestFactoryParser(method);
  parser.parseMethodAnnotations(responseType);
  parser.parseParameters(retrofit);
  return parser.toRequestFactory(retrofit.baseUrl());
}

The constructor and the conversion methods do nothing special, so let's focus on the two parsing methods. The former, parseMethodAnnotations(), is responsible to scan the annotations of a method, similar to what we have seen here, and to delegate to parseHttpMethodAndPath() the setting of the method type (e.g. get, put, post, delete, etc) and the parsing of the path parameters (e.g. "user" in the url as above).

private void parseMethodAnnotations(Type responseType) {
  for (Annotation annotation : method.getAnnotations()) {
    if (annotation instanceof DELETE) {
      parseHttpMethodAndPath("DELETE", ((DELETE) annotation).value(), false);
    } else if (annotation instanceof GET) {
      parseHttpMethodAndPath("GET", ((GET) annotation).value(), false);
    } // ...
  }
  // some checks here
}

The latter method, parseParameter() is really long (300+ lines) but not so complicated. It parses parameters annotations, not method's, and creates the corresponing RequestActions triggered in the RequestFactory.

Only a couple of points to note here: firstly, @Body, @Part and @PartMap are the only actions that use a Converter to RequestBody instead of to String, see the second type parameter. Secondly, Retrofit supports array/varargs and iterable notations to pass headers, query parameters, form fields and body parts, see the docs for the corresponding annotations. This is implemented by the methods RequestAction.array() and RequestAction.iterable(): here the action will wrap itself in another action which will loop through all the values and perform the original action on each value.

Plumbing: Calls, call adapters and converters

In this section we talk about the plumming logic of Calls, CallAdapters and Converters. A Call object encapsulates a pending request that can be sent either synchronously, with execute(), or asynchronously, with enqueue(Callback<T>). In the former case, a Response<T> is returned in the return value (with T parametrized with the type of the wanted resource) or an Exception is raised in case of failure. In the latter case, a Callback<T> object is passed in as argument (with T still parametrized with the type of the wanted resource) and we get a callback either onSuccess(Response<T>) or onFailure(Throwable).

Call objects are new to Retrofit 2 and provide a handy interface for manipulating requests, for example to clone them or to stash them away for later use. Furthermore, the returned Response objects are useful to obtain both the body of the response, usually as some user defined entity converted by some Json library, and additional information such as the response status code or errors.

How are Call objects created? This is done every time MethodHandler is invoked:

Object invoke(Object... args) {
  return callAdapter.adapt(
    new OkHttpCall<>(callFactory, requestFactory, args, responseConverter));
}

The OkHttpCall class is a special implementation that adapts a okhttp Call to a Retrofit Call. It has three main responsabilities:

  1. Create raw okhttp3.Calls, using an OkHttpClient (the call factory).
  2. Parse back the response with a Converter in parseResponse().
  3. Adapt the okhttp3.Call API to the retrofit2.Call API (e.g. forwarding cancellation and translating callbacks)

Now, you might have noticed that the OkHttpCall is not returned directly but "adapted" through a call adapter. This might seem superflous but it is handy to have a flexible API to get responses back as custom wrappers. For example, RxJava fans could get back responses as Observable<T> instead of Call<T>; see also CustomCallAdapter in the samples.

Call adapters have another usage though, namely threading: they are also used to invoke callbacks on a specific thread. This is fundamental on Android where success/failure callbacks run on the main thread. To see this, let's dive into the ExecutorCallAdapterFactory.ExecutorCallbackCall code, in the enqueue() method:

// ...
final Call<T> delegate;
// ...

@Override public void enqueue(final Callback<T> callback) {
   delegate.enqueue(new Callback<T>() {
     @Override public void onResponse(final Response<T> response) {
       callbackExecutor.execute(new Runnable() {
         @Override public void run() {
           if (delegate.isCanceled()) {
             // Emulate OkHttp's behavior of throwing/delivering an 
             // IOException on cancelation
             callback.onFailure(new IOException("Canceled"));
           } else {
             callback.onResponse(response);
           }
         }
       });
     }

     @Override public void onFailure(final Throwable t) {
       callbackExecutor.execute(new Runnable() {
         @Override public void run() {
           callback.onFailure(t);
         }
       });
     }
   });
 }

The delegate here creates new callbacks to execute the real callbacks on the given executor. This looks like the callbacks were jumping to the requested thread: for example, on Android the executor will be MainThreadExecutor which will post() the real callbacks on the main thread.

Call adapters can be set on the Retrofit buider calling Retrofit.Builder.addCallAdapterFactory(): when it's time to locate one, i.e. when the MethodHandler is first created, all the factories will be scanned in order, and the first that matches the given return type (e.g. Call) will be used. See the methods Retrofit.callAdapter() and Retrofit.nextCallAdapter() for more detail.

Finally let's take a look to Converters. They come in three flavors:

  1. Response converters, i.e. Converter<ResponseBody, T>: used by OkHttpCall<T> to convert a okhttp3.ResponseBody to T.
  2. Request converters, i.e. Converter<T, RequestBody>: used in RequestAction.Body, RequestAction.Part and RequestAction.PartMap when converting a user defined pojo to a RequestBody to be passed in the RequestBuilder.
  3. String converter, i.e. Converter<T, String>: used in all other RequestActions, for example to convert a @Path annotated argument to a string.

It must be mentioned that the default string converter is ToStringConverter that just call toString() on the given argument; if the argument were a primitive the dynamic proxy would wrap it. All three types are added through the Retrofit builder's method addConverterFactory().