Relight is a library to create and view relightable images (RTI).

Demo

Features

Relight supports:

• [Relight](#Relight fitter) a PCA based format (see paper)
• PTM, HSH
• Web viewer
• Zoomify, deepzoom, google, IIP, IIIF

Relight new formats provide better accuracy and smaller size.

Relight fitter

relight-cli is a Qt command-line program to process a stack of photos into an RTI.

Usage: relight-cli [-frqpsScCey]<input folder> [output folder]

input folder containing a .lp with number of photos and light directions
optional output folder (default is ./)

-b <basis>: rbf(default), ptm, lptm, hsh, yrbf, bilinear
-p <int>  : number of planes (default: 9)
-y <int>  : number of Y planes in YCC
-r <int>  : side of the basis bilinear grid (default 8)
-q <int>  : jpeg quality (default: 90)
-s <int>  : sampling rate for pca (default 4)
-S <int>  : sigma in rgf gaussian interpolation default 0.125 (~100 img)
-C        : apply chroma subsampling 
-e        : evaluate reconstruction error (default: false)

relight-cli can also be used to convert .ptm files into relight format:

relight-cli [-q]<file.ptm> [output folder]
-q <int>  : jpeg quality (default: 90)

Relight web format

Relight format is a directory containing an info.json file and a few images.

Json contains the following fields:

• width:
• height:
• format: image extension: .jpg, .png etc.
• type: one of lptm, hsh, rbf, bilinear ycc
• nplanes: numer of coefficient planes
• resolution: for bilinear types the side of the bilinear grid
• colorspace: one of lrgb, rgb, ycc, mrgb, mycc
• sigma: rbf interpolation parameter
• lights: mandatory for rbf interpolation
• materials: an array of materials, each material specify scale, bias and range
• scale: an array of floats
• bias: an array of floats
• range: an array of floats
• basis: an array of unsigned chars containing the basis for rbf, bilinear and ycc basis.

Each image contains 3 coefficient planes

• PTM: r, g, b, 1, u, v, u^2, uv, v^2
• HSH: see source code :)
• RBF: PCA basis coefficients
• BILINEAR: PCA basis coefficients
• YCC: PCA basis coefficients as Y₀C₀C₀, Y₁C₁C₁... Y_kC_kC_k, Y_k+1Y_k+2Y_k+3

Scale and bias will be applied to the texture coefficients (from [0, 1]) as:

(c - bias)*scale

Relight tiled web format

relight.js support a variety of tiled formats

• Deepzoom
• Zoomify
• Google maps
• IIP
• IIIF

libvips can be used to generate deepzoom, zoomify and google pyramidal formats, scripts can be found in the directory test.

Web Viewer

relight.min.js is a small Javascript library to render the RTI on a WebGL canvas.

var relight = new Relight(canvas, options);

Options:

• url: path to a directory containing the web format
• layout: deepzoom, zoomify, google, iip, iiif or image for the web format
• server: server URL for use with iip or iiif
• stack: true or false - whether image is an image stack (IIP only)
• light: initial light, array x, y, z.
• pos: initial view object { x: y: z:}
• border: for tiled formats amount of prefetching tiles arount the view (default 1)
• fit: scale and center the view on load

Members:

• pos: the position of the view as {x: y: z: t} where x and y are the coords of the center of the screen in full scale image. z is the zoom level with 0 being the original image. t is for interpolation in ms
• light: array of x, y, z light direction
• onload: function to be called when rendering is ready

Methods:

• setUrl(url): change url
• resize(width, height): resize canvas
• zoom(amount, dt):
• pan(dx, dy, dt): change the center of the view
• setPosition(x, y, z, dt):
• center(dt): center view (but does not change zoom
• centerAndScale(dt): fit the view to the canvas
• setLight(x, y, z): change light direction
• draw(time): draw the canvas, use time for interpolation
• redraw(): schedule an animaterequest

Building relight

Debian Linux

$ uname -a
Linux x220 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) x86_64 GNU/Linux

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Install dependencies. (replace libjpeg62-turbo0dev with libturbojpeg-dev in Ubuntu 2020)

$ apt update && apt install \
    build-essential \
    cmake \
    git \
    qtbase5-dev \
    libeigen3-dev \
    libjpeg62-turbo-dev \
    libomp-dev

Clone this repository and build.

$ git clone https://github.com/cnr-isti-vclab/relight.git
$ cd relight
$ git submodule update --init --recursive
$ cmake .
$ make

TODO

• White balance and other conversion from RAW features (dcraw)
• use color tablet to calibrate raw images
• contrast and other image processing
• measure to be added to the images from tag
• remove lens distortion
• find spheres (very optional)
• mask artifact
• crop
• join pieces
• align images using mutual information (or better an edge detector?)
• find highlight