Overview
This is another post on Spring Boot that will show how to build a sample web application using Deep Java Library (DJL), an open-source Deep Learning library for Java to diagnose COVID-19 on X-ray images.
The sample app is a Spring Boot based version of DJL’s similar COVID-19 example and it has a simple static HTML page built using Twitter Bootstrap and JQuery where users can submit an image URL to a REST api where the DJL library will download the image and predict if it’s an X-ray image of lungs infected with COVID-19 or not.
The link to the source code is included at the end of this post.
Disclaimer: this is only a demo application based on the dataset at https://github.com/ieee8023/covid-chestxray-dataset and it SHOULD NOT be used for actual medical diagnosis.
Deep Java Library
As mentioned earlier, DJL is a Java-based library that supports multiple
Deep Learning frameworks like Apache MxNet, PyTorch and Tensorflow. Since most Deep Learning engines are built using Python and not in Java, DJL built engine adapters to access each of these engines’ native shared library.
DJL does it in an elegant way making it dead simple to switch from one framework to the other depending on the use case.
Dependencies
The app needs the Spring Boot web starter:
<code> <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency></code>
And the commons-io library for some basic I/O operations:
<code> <dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency></code>
The Lombok library, too, as I’m too lazy to write the getters and setters:
<code> <dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency></code>
And finally the DJL dependencies for this sample app:
<code> <dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>${ai.djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.tensorflow</groupId>
<artifactId>tensorflow-api</artifactId>
<version>${ai.djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.tensorflow</groupId>
<artifactId>tensorflow-engine</artifactId>
<version>${ai.djl.version}</version>
</dependency>
<dependency>
<groupId>ai.djl.tensorflow</groupId>
<artifactId>tensorflow-native-auto</artifactId>
<version>${tensorflow-native-auto.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>${jna.version}</version> <!-- overrides default spring boot version to comply with DJL -->
</dependency></code>
Here’s the list of Maven properties needed for the DJL dependency versions:
<code> <properties>
<java.version>1.8</java.version>
<ai.djl.version>0.5.0</ai.djl.version>
<jna.version>5.3.0</jna.version>
<tensorflow-native-auto.version>2.1.0</tensorflow-native-auto.version>
</properties></code>
XRayApplication class
This class’s main() method will fire up the Spring Boot application and it looks like most other Application class files:
<code>@SpringBootApplication
public class XRayApplication {
public static void main(String[] args) {
SpringApplication.run(XRayApplication.class, args);
}
}</code>
Configuration
In order to configure the DJL library, let’s create a DjlConfig class with the @Configuration annotation.
This class will define a ZooModel Spring Bean that will help predicting if the submitted image URL belongs to a COVID-19 infected lung:
<code> @Bean
public ZooModel xrayModel() throws Exception {
Criteria<BufferedImage, Classifications> criteria =
Criteria.builder()
.setTypes(BufferedImage.class, Classifications.class)
.optTranslator(new XrayTranslator())
.build();
return ModelZoo.loadModel(criteria);
}</code>
What this code says is that we create a ZooModel object with a BufferedImage input and Classifications (more on that later) output type and it uses an XrayTranslator object to transform the input images to a format needed by the Deep Learning model to function properly.
Here’s the code for the XrayTranslator which is an inner class within DjlConfig:
<code> public static final class XrayTranslator implements Translator<BufferedImage, Classifications> {
private static final List<String> CLASSES = Arrays.asList("covid-19", "normal");
@Override
public NDList processInput(TranslatorContext ctx, BufferedImage input) {
NDArray array =
BufferedImageUtils.toNDArray(
ctx.getNDManager(), input, NDImageUtils.Flag.COLOR);
array = NDImageUtils.resize(array, 224).div(255.0f);
return new NDList(array);
}
@Override
public Classifications processOutput(TranslatorContext ctx, NDList list) {
NDArray probabilities = list.singletonOrThrow();
return new Classifications(CLASSES, probabilities);
}
}
</code>
Covid19Service
The Covid19Service class will handle the business logic to diagnose the X-ray images and as you’ll see, surprisingly, it’s really just few lines of code:
<code>@Service
public class Covid19Service {
@Autowired
private ZooModel xrayModel;
public String diagnose(String imageUrl) {
try (Predictor<BufferedImage, Classifications> predictor = xrayModel.newPredictor()) {
Classifications result = predictor.predict(BufferedImageUtils.fromUrl(imageUrl));
return "Diagnose: "
+ result.best().getClassName()
+ " , probability: "
+ result.best().getProbability();
} catch (Exception e) {
throw new RuntimeException("Failed to diagnose", e);
}
}
}</code>
The ZooModel bean created in the DjlConfig class is autowired and used in the diagnose()method that has an imageUrl parameter.
Within the method we create a Predictorobject using the try-resource block (as the predictor needs to be closed after execution) and use it to run the BufferedImage(created using the imageUrl parameter) through a pre-trained Tensorflow model.
For more details on the model visit: https://www.pyimagesearch.com/2020/03/16/detecting-covid-19-in-x-ray-images-with-keras-tensorflow-and-deep-learning/.
Once the diagnose()method is run, the Classificationsresult object will show if the lungs on the X-ray image were infected with COVID-19 or not and with what probability.
Covid19Controller
This controller class defines the REST api to diagnose X-ray images which will be consumed by our simple front-end app:
<code>@RestController
@RequestMapping(value = "/api/v1", produces = MediaType.APPLICATION_JSON_VALUE)
public class Covid19Controller {
private final Covid19Service covid19Service;
public Covid19Controller(Covid19Service covid19Service) {
this.covid19Service = covid19Service;
}
@GetMapping("/covid19/diagnose")
public ResponseEntity diagnose(@RequestParam String imageUrl) {
String answer = covid19Service.diagnose(imageUrl);
return ResponseEntity.ok(Collections.singletonMap("result", answer));
}
}</code>
The @RestControllerannotation tells Spring that in our MVC design this is a Controller bean that defines the REST api
The @RequestMappingannotation tells Spring that paths of all REST apis within this class should be prefixed with /api/v1and all REST apis will return application\jsonresponses.
The Covid19Servicethat we discussed earlier is autowired in the constructor and later on used by the diagnose REST api at the GET /api/v1/covid19/diagnosepath.
The diagnose api takes an imageUrlrequest param and returns a JSON document with the String representation of the results.
Front-end
The Spring Boot app has a simple static index.html file as a front-end client for the diagnose REST api and it uses Twitter Bootstrap for the responsive design and JQuery to make the REST api call:
<code><head>
<link rel="stylesheet" href="/css/bootstrap.min.css"/>
<script src="/js/jquery.min.js"></script>
</head></code>
The file has an HTML form that can capture an X-ray image URL from the user:
<code> <form id="diagnoseForm" class="mb-4">
<div class="input-group">
<input type="url" id="imageUrl" class="form-control" required
placeholder="Enter a image url"
aria-label="Image URL">
<div class="input-group-append">
<button class="btn btn-outline-primary">Submit</button>
</div>
</div>
</form></code>
Once the form is submitted, the REST api may take a while to respond. In the meantime the page will show a spinner and once the response is received, the text will be displayed within the diagnose div:
<code> <div class="row ml-1">
<div id="spinnerDiagnose" class="text-primary" role="status">
<span class="sr-only">Loading...</span>
</div>
<div id="diagnose"></div>
</div></code>
See below the javascript code:
<code>$( "#diagnoseForm" ).submit(function( event ) {
const imageUrl = $('#imageUrl').val();
$('#spinnerDiagnose').addClass('spinner-border');
$('#diagnose').html('');
$.ajax('/api/v1/covid19/diagnose?imageUrl='+imageUrl)
.done(data => {
$('#spinnerDiagnose').removeClass('spinner-border');
$('#diagnose').html(data.result);
})
.fail(err => {
$('#spinnerDiagnose').removeClass('spinner-border');
$('#diagnose').html('Failed to get answer');
});
event.preventDefault();
});</code>
When the form’s submit event is triggered, the code gets the imageUrl value, show’s the spinner, clears the content of the diagnose div from previous runs and calls the diagnose REST api with the imageUrl.
In case of a successful response the code hides the spinner and displays the results within the diagnosediv.
In case of an error the code also hides the spinner and displays a generic error message.
Running the app
The app needs Tensorflow to be downloaded first before it can be run.
Run the following command in the projects root folder:
<code>mkdir models cd models curl https://djl-tensorflow-javacpp.s3.amazonaws.com/tensorflow-models/covid-19/saved_model.zip | jar xv cd .. ./mvnw spring-boot:run -Dai.djl.repository.zoo.location=models/saved_model</code>
Then visit http://localhost:8080/index.html to get diagnose on X-ray image URLs. Sample images to use:
Overview
In this tutorial we reviewed how to create a sample Deep Learning Java app using Spring Boot, DJL and Tensorflow.
The source code for the post is available at https://github.com/davidkiss/djl-spring-boot-xray.