Additional Operators for Image Models

Implements additional operators for computer vision models, including operators necessary for image segmentation and object detection deep learning models.

torchvisionlib

The goal of torchvisionlib is to provide access to C++ operations implemented in torchvision. It provides plain R acesss to some of those C++ operations but, most importantly it provides full support for JIT operators defined in torchvision, allowing us to load ‘scripted’ object detection and image segmentation models.

Installation

torchvisionlib can be installed from CRAN with:

install.packages("torchvisionlib")

You can also install the development version of torchvisionlib from GitHub with:

# install.packages("devtools")
devtools::install_github("mlverse/torchvisionlib")

Example

Suppose that we want to load an image detection model implemented in torchvision. First, in Python, we can save JIT script and then save this model:

import torch
import torchvision

model = torchvision.models.detection.fasterrcnn_mobilenet_v3_large_320_fpn(pretrained=True)
model.eval()

jit_model = torch.jit.script(model)
torch.jit.save(jit_model, "fasterrcnn_mobilenet_v3_large_320_fpn.pt")

We can then load this model in R. Simply loading torchvisionlib will register all JIT operators, and we can use torch::jit_load().

library(torchvisionlib)
model <- torch::jit_load("fasterrcnn_mobilenet_v3_large_320_fpn.pt")
model
#> An `nn_module` containing 19,386,354 parameters.
#> 
#> ── Modules ─────────────────────────────────────────────────────────────────────
#> • transform: <script_module> #0 parameters
#> • backbone: <script_module> #4,414,944 parameters
#> • rpn: <script_module> #609,355 parameters
#> • roi_heads: <script_module> #14,362,055 parameters

You can then use this model to make preditions or even fine tuning.