MORPHOLO C++ Library

User's Guide

for glasses-free multi-view stereo vision

and streaming of live 3D video

version 1.0 - Dec 2019

by Enrique Canessa & Livio Tenze

www.morpholo.it

Table of Contents

1.1Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4Credits and Contacts . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1Stereo Image Capture . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2Morphing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.1Computer with Linux O.S. and Slanted Lenticular Display . . . . 11

3.2Stereo Camera with Parallel or Convergent Lens . . . . . . . . . . 11

4.1Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.2Library Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4.3Uninstall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.1Extract Calibration Data from 3D Display . . . . . . . . . . . . . 16

5.2Create LUT Mapping from Calibration File and Quilt Inputs . . . 16

5.3.2Quilt from single stereo camera with parallel lens . . . . . 18

5.3.3Quilt from two different parallel or convergent (toe-in) camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5.4Generate Native Image / Multi-Vision Stereo Vision . . . . . . . . 18

5.4.1 Native image from sorted views stored in a directory . . . 18

5.4.2Display Native image from (historical) stereo L-R images . 19

6.2Elapsed Processing Time . . . . . . . . . . . . . . . . . . . . . . . 23

6.33D Real-Time Video Streaming . . . . . . . . . . . . . . . . . . . 25

1

6.3.1Using single stereo camera with parallel lens . . . . . . . . 25

6.3.2Using two different parallel or convergent (toe-in) camera . 26

About this Tutorial

This User's Guide was written for the benefit of all teachers, students and all those interested in wanting to use the MORPHOLO C++ Libray for studies, research, demos, tests, etc. on easy and low-cost 3D Vision. The contents in this Guide are free for public use.

2

1SECTION

About MORPHOLO

The MORPHOLO C++ extended Library allows to convert a specific stereoscopic snapshot into a Native multi-view image through morphing algorithms taking into account display calibration data for specific slanted lenticular 3D monitors. A slanted lenticular system allows to distribute loss of resolution in both directions, the horizontal and vertical, by slanting the structure of the lenticular lens.

Figure 1: Slanted lenticular screen on a LCD array

used to enhance image quality [1].

The MORPHOLO Library offers a fast conversion algotithm to transform stereoscopic images into reasonable holograms via the automatic implementation of a one-time configu- ration Lookup Table (LUT) [2] -to replace runtime computation, and by the unsupervised morphing deformations between Left (L) and Right (R) stereoscopic images (or from a given sequential set of plain images).

Figure 2: Typical L-R Stereoscopic images.

Although morphing cannot provide complete depth information on distant regions, the generation of multi-view images starting from a pair of stereoscopic images through MOR- PHOLO could still offer a potential alternative method for 3D vision. It mainly extrapolates the parallax encoded in the steroscopic images by retrieving information from nearby pixels only!

3

MORPHOLO can also be implemented for glasses-free live applicatons of 3D video stream- ing, and for diverse innovative scientific, engineering and 3D video game applications. In particular, it could be also useful for Panoptic studies of dynamics 3D structures, and Stere- omicroscopy analysis in real-time for microsurgery or for examining microscopic samples with complex 3D topography.

1.1Motivation

A 2D image is worth a thousand words. Notwithstanding a 3D reconstruction of a scene is worth a million [1].

Multi-views 3D display technology is still under development even if the 3D TV technology was emerging together with the raising of the XXI century. It still needs to be optimized so as to reduce the high production costs and some of the health concerns induced by such 3D presentations.

The new class of slanted lenticular 3D displays, such as the standalone HDMI Looking Glass HoloPlay Monitors (www.lookingglassfactory.com), allow to display a hologram of simultaneous N × M different views at 60fps formed through a collage of images (or Quilt as in Fig.8) without the need of wearing unnatural headsets. These devices combine light-field and volumetric technologies, and allow to visualize the complexity of real world objects and 3D scenarios for around 40to 50fields of view (FOV). These 3D screens have specific computer display calibration values each for a correct rendering.

Motivated by these novel technologies, and by an earlier study on animating historical stereograms with optical flow morphing [3], we have developed the MORPHOLO Library to reconstruct reality from stereoscopic images. The Library is wrtten in C++ and imple- mented using the openCV Library with the goal of minimizing the image processing time to approach real time applications in 3D streaming using low-cost hardware (i.e., simple DIY or available stereo webcams) without the need for the viewer to wear any special 3D glasses.

Full technical description of the MORPHOLO project including classes, structs, unions and interfaces with brief descriptions, can be found at www.morpholo.it or downloading the ’.deb’ package.

4

Figure 3: Class List of MORPHOLO Library.

1.2Features

MORPHOLO command lines lead to extend observations of 2D stereoscopic images (or, alternatively, from a given sequential set of plain images) virtual interpretations of reality –at least, within a wide FOV.

In other words, MORPHOLO C++ Library provides the following features:

Communication with the slanted lenticular PC screen to get its calibration data.

(i)To read display calibration values via a specific eeprom interface [4] (i.e., read- /write ’.json’ file).

(ii)To generate an associated LUT from these values and also according to some given resolution for the screen, FOV, and other variables in the ’.json’ file.

The transformation of a stereoscopic image to a collection of views by morphing (that all together could form a single 3D video frame).

(iii)Retrieve synchronous stereoscopic images taken from a stereo webcam.

(iv)Generate n intermediate views by morphing.

(v)Place these views into a single standard Quilt image.

The mapping of these multi-views to the destination holographic space.

(vi)Treat the collection of views in the Quilt into Native images consisting of light- fields that can be correctly displayed on the specific slanted, lenticular screen.

1.3Copyright

c Permission to use, copy, and distribute the MORPHOLO C++ Library and its doc- umentation for educational purposes ONLY, and without any fee, is hereby granted provided that reference to www.morpholo.it appears in all distributed copies and in all supporting documentation. This Library is provided ”as is” without any express or implied warranty.

5

1.4Credits and Contacts

MORPHOLO C++ Library for Morphing Holograms is developed and maintained by En- rique Canessa and Livio Tenze from the ICTP Science Dissemination Unit (SDU), in Trieste, Italy.

For further information, get binaries, papers, presentations, manuals, etc., or to report Bugs, please contact us at: ’sdu@ictp.it’ or visit our project website: www.morpholo.it

6

2SECTION

Theory

2.1Stereo Image Capture

Lenticular, autostereoscopic multi-view displays require multiple views of a scene to provide motion parallax and get a realistic 3D experience. By changing the viewing angle, different stereoscopic pairs are perceived. However, capturing arbitrary number of views can be cumbersome, and in some occasions impossible due., e.g., to occlusion [5].

Instant stereoscopic images can be taken using a set of parallel and toe-in cameras horizon- tally separated as depicted in the next figure [6]. Convergence avoids cutting off lateral parts of the two images to fulfill frame images in stereo display. In the parallel-cameras method, the visual camera axes are parallel, making the convergence distance z to be infinite.

Stereo images captured with toe-in cameras may result in geometric distortion due to the projection difference. Vertical parallax can be introduced by converged cameras causing a change in the vertical disparities at the viewer’s retinas and are likely to affect the 3D perception [7]. Perspective correction and alignment are done during computer graphic rendering [8].

Figure 4: Capture configurations for stereo systems with a horizontal baseline: Top view of Parallel cameras (left) and Converging cameras, also called toe-in.

7

Notwithstanding such constraints, we have verified via trial-and-error approach that [9]:

using slightly toe-in cameras, they actually produce a 3D image that has more depth as compared to using paralled cameras images. This is so when the Native 3D image produced by MORPHOLO is visualized in slanted lenticular LCD displays (such as the Looking Glass Holoplay).

In fact, slanted lenticular-based stereoscopic 3D displays provide only horizontal parallax [1]. They generally lack vertical parallax, hence any vertical parallax introduced using stereo toe-in cameras gets reduced.

The projection convergence allows to create the illusion of a captured multi-view scene having much more 3-dimensional depth.

2.2Morphing Algorithms

Within MORPHOLO, one can automatically generate a given number of intermediate views (having different viewing angles and needed to create native holographic frames from Quilts, by selecting from two different morphing techniques: (i) Disparity Map and (ii) DeepFlow. As explained below, View Morphing has not been included since it presents limits for real-time 3D streaming. Same as for tri-view morphing which follow the view morphing approach as in [10].

Disparity Morphing: A method that depends on the estimated disparity map for each image pair based on the epipolar geometry [11, 12].

The epipolar constraints speed up the triangulation of the corresponding reference points in the L and R image. The non-linear forward and backward morphing func- tions are defined based on the estimated disparity maps with some interpolation function pre-defined to deal with occlusion. For smaller disparity, the darker a point is and further away it is from the camera.

Figure 5: Disparity Map from stereo images with a color coding.

DeepFlow: A variational matching approach for optical flow [13].

Optical flow computation is a key component in many computer vision systems de- signed for tasks such as action detection or activity recognition overcoming problems that arise in realistic videos such as: motion discontinuities, occlusions, illumination changes and ability to deal with (large) displacements.

8

Figure 6: Optical flow, or vector field, describing movement

between stereoscopic images.

View Morphing: An extension of existing image morphing tools for transitions be- tween images by adding image prewarping and postwarping steps [14].

It can handle 3D projective camera and scene transformations, creating more natural transitions between images at the expense of long execution time. This algorithm is both complex and slow, making it computational extensive for real-time 3D video streaming. This is because of the difficulty of encapsulating motion with certain number of keypoints or matching correspondences (given manually or predicted by deep learning algorithms), during the dynamics of a scene.

Figure 7: Example of point correspondences in steroscopic images, and

MORPHOLO Native output on a Holoplay using View Morphing.

2.3The Quilt

The intermediate views generated using MORPHOLO can be then added into a Quilt collage sequentially as shown below. The fading out from L to R images, aims to render a whole panoramic scene into just a given number of in-between snapshots.

Figure 8: Central part of a 9x5 Quilt generated by morphing

starting from the stereoscopic images in Fig.2.

9

The CPU time for creating such sets of images, and especially the holographic multi-view outputs (i.e., figure 4), varies considerably between diffrent morphing algorithms. Key pro- cessing steps in the DeepFlow field include the arbitrary, and time-consuming, matching by polynomial interpolation to approximate neighborhood pixel intensities, warping and opti- mization. Optical flow techniques are sensitive to the presence of occlusions, illumination variations and out-of-plane movements, whereas disparity map leads to obtain smoother and faster translation motion between consecutive images.

However, according to our experiences:

DeepFlow morphing alghorithm leads by far to a better and acceptable illusion of depth and parallax in the horizontal direction when using toe-in webcams.

The N × M morphed images in this figure –forming the Quilt, are converted into a native light-field image via Eq.(1) as shown in the figure below.

2.4Light-Fields and Native Images

A generic expression for the relation between the pixels of a slanted lenticular 3D LCD and the multiple perspective views was first derived in [15]. Each sub-pixel on the 3D-LCD is mapped to a certain view number and color value (i.e., in the lightfield domain). If i and j denote the panel coordinates for each sub-pixel, then

where N denotes the view number of a certain viewpoint, α the slanted angle between the lenticular lens and the LCD panel and Px the lenticular pitch. Upscaling multiple views (e.g., upscaling from the Quilt to the Native Looking Glass image) requires lots of CPU resources and increases system complexity [16, 17, 18]. With the generation of a device-dependent LUT table this time can be reduced considerably.

Figure 9: Example of multi-view hologram output by MORPHOLO Library

starting from morphed stereoscopic images. See also video demo at: https: // www. youtube. com/ watch? v= 6FAhmI-vtLQ

MORPHOLO C++ Library creates multiview images via Eq.(1) starting from a morphed stereoscopic scene (or, alternatively, from a given set of sequential photos) to form the rich Native light-field image.

10

3SECTION

Technical Notes /

Requirements

The simplest (DIY or available) harware needed so far to implement MORPHOLO is as follows:

3.1Computer with Linux O.S. and Slanted Lenticular Display

The PC used were standard Intel Core (i3-i5, 64bit, 4-8G RAM), 1366x768p resolu- tion, with graphics card Nvidia GeoForce 820M output 2560x1600p and O.S. Linux Ubuntu 19.04.

The Looking Glass HoloPlay in Fig.10 is an external HDMI video monitor with slanted lenticular display. It provides a novel glasses-free way to preview 3D objects and scenes within a FOV 40to 50. Each Looking Glass holds own calibration (’.json’) data for correct rendering.

Figure 10: Headset-free Looking Glass 3D Holoplay monitor.

www.lookingglassfactory. com

3.2Stereo Camera with Parallel or Convergent Lens

Dual lens USB 3.0 camera: A compact ELP-960P2CAM stereo webcam with no distortion dual lens aligned parallel as in Fig.11. Its two camera video frames are synchronous with 1280x960p max. resolution with a large FOV. This feature enables us to observe by one sigle shot a scene from two different viewpoints without the need for any prior (L-R images) calibration.

11

Figure 11: Compact, low-cost ELP synchronized stereo webcam.

www. lookingglassfactory. com

Or, alternatively,

Building the stereo camera by two modular USB webcam in parallel. These need to be aligned, calibrated, etc at different separations and the video from both perspectives are retrieved in two separate windows. To get started with this see for example [19].

Figure 12: Pair of modular webcams in the Holoplay display.

Or, alternatively,

One modular USB webcam only with mirrors. Mirrors can be used to convert a single camera into a pseudo-stereo imaging system to view and record an object or scene from two different angles, one in each half of the CMOS camera sensor [20, 21]. 3D systems with mirror and prisms are extensively used for 3D macro photography [22].

Figure 13: Single camera stereo systems using two and four planar mirrors [21].

12

Or, alternatively,

Two modular USB webcam convergent or toe-in (to improve the 3D) at close sepa- ration.

Figure 14: Convergent DIY stereo webcam with FOV of 120each lens and toe-in angle of 5. Graphics of the OpenSCAD ’.stl’ file used for the holder (right).

As mentioned before, we have verified that slightly toe-in cameras produce a 3D image that has more depth as compared to using paralled cameras images. This is so when the Native 3D image produced by MORPHOLO is visualized in slanted lenticular LCD displays since vertical parallax gets reduced [1]. A toe-in angle of 5between the two webcam is enough for a FOV of 120each with lens baseline of 4 cm (about 38% smaller than the human eyes separation).

13

4SECTION

How to Install

morpholo-x.x.x-Linux.deb

The latest version of the Debian morpholo-x.x.x-Linux.deb package can be downloaded from www.morpholo.it A connection to the Internet is needed to download new releases.

4.1Dependencies

It is necessary to install first some extra packages and their dependencies. The following packages (and their dependencies) are needed to be installed before hand:

libopencv-videoio3.2 libopencv-imgproc3.2 libopencv-core3.2 libopencv- imgcodecs3.2 libopencv-contrib3.2 libopencv-calib3d3.2 libopencv-imgcodecs3.2 libopencv-video3.2 libopencv-flann3.2 libqt5widgets5 libqt5gui5 libqt5core5a libhidapi-libusb0

These could become out of date, so check the list of needed packages by issuing the following command:

dpkg -I morpholo-x.x.x-Linux.deb

In ’Depends: · · · ’ you can find the updated list of required packages.

To install the required packages (listed above) issue the command: ’sudo apt-get install <pkg1> <pkg2> · · · ’ and so on. For example,

sudo apt-get install libopencv-videoio3.2 libopencv-imgproc3.2 · · ·

4.2Library Install

To install the MORPHOLO (’.deb’) package in the /opt/morpholo directory type the command line:

sudo dpkg -i morpholo-x.x.x-Linux.deb

In case some ’warning: files list file for package · · · ’ may appear on the screen after having issued ’dpkg’, then type

sudo apt-get install -f

which install what is missing to ’dpkg’.

It also necessary to set the path to the working directory before the MORPHOLO com- mands can be used. Then, type:

14

sudo -i

cd /opt/morpholo/bin/

source /opt/morpholo/bin/setupvars.sh

which defines

INSTALLDIR=”/opt/morpholo”

export PATH=$INSTALLDIR/bin:${PATH}

export LD LIBRARY PATH=$INSTALLDIR/lib:${LD LIBRARY PATH}

4.3Uninstall

In order to remove the MORPHOLO (’.deb’) package type

sudo dpkg -r morpholo

Then check that the /opt/morpholo directory becomes empty!

15

5SECTION

Using MORPHOLO

5.1Extract Calibration Data from 3D Display

Each Looking Glass HoloPlay (shown in Fig.10) holds its own per-device calibration data set in the phase of manufacturing for correct rendering! To get this data (in a ’.json’-type of text file), connect the HoloPlay device to your PC and issue the command as root:

sudo ./morph2native -m DEFAULT MAP FILE -c CALIBRATION FILE

with

This command generates two files: (i) one with a default LUT mapping (that can later be also re-estimated), and (ii) another with the unique calibration data for the particular HoloPlay being used.

For example, the above command can look like this:

./morph2native -m default map.map mycal.json

The CALIBRATION FILE (i.e., ’mycal.json’ in the example) will look like this:

”configVersion”:”1.0”,”serial”:”LKG-2K-02491”,”pitch”:”value”:47.56159591674805, ”slope”:”value”:-5.5113043785095219,”center”:”value”:-0.09782609343528748, ”view- Cone”:”value”:40.0,”invView”:”value”:1.0,”verticalAngle”:”value”:0.0,”DPI”:”value”: 338.0,”screenW”:”value”:2560.0,”screenH”:”value”:1600.0,”flipImageX”:”value”:0.0, ”flipImageY”:”value”:0.0,”flipSubp”:”value”:0.0

The binary DEFAULT MAP FILE file contains the LUT mapping for a default 8x4 Quilt whose single image is assumed to be size 256 × 512px. As shown next this ’.map’ file can be modified for different Quilt sizes, adapted to the CALIBRATION FILE file.

5.2Create LUT Mapping from Calibration File and Quilt Inputs

Use:

16

./morph2native -r RESOLUTION -q NxM QUILT

-m FINAL MAP FILE CALIBRATION FILE

with

For example, the above command can look like this:

./morph2native -r 240x320 -q 9x5 -m mymap.map mycal.json

5.3Generate a Quilt

Up scaling multiple views from the stereo L-R images to the Native image can require sub- stantial computer resources depending on the Quilt resolution. Each point of these Quilts has to be mapped to the destination Native holographic image of 2560x1600p. The mapping procedure needs to be very accurate according to the Quilt dimensions and has to take into account the specific display calibration data. Within MORPHOLO, it is possible to get the Quilt from different sources and by selecting from two different morphing techniques: (i) Disparity Map and (ii) DeepFlow.

5.3.1Quilt from stereo L-R images

morph2quilt is a Disparity map- and DeepFlow-based tool to generate intermediate views and the Quilt, using archived left and right stereoscopic images.

./morph2quilt -l IMG1 -r IMG2 -m NxM QUILT -d -s <factor> QUILT FILE

with

By default, Disparity map is used for the morphing, otherwise use the option ’-d’ to apply the DeepFlow algorthim. The option ’-s’ sets a subsampling in the morphing process to speed up calculations, e.g., use small values ’-s 2’.

For example, to create a Quilt by DeepFlow the above command can look like this:

./morph2quilt -l photo2.png -r photo4.png -m 9x5 -d myquilt stereo imgs.png

17

5.3.2Quilt from single stereo camera with parallel lens

A stereo camera with no distortion dual lens, aligned parallel as in Fig.11, together with a single modular USB webcam with mirrors as in Fig.13, can be used as a pseudo-stereo imaging system. In this case, you can use the standard applications ffmpeg and convert to capture images. For example, use:

ffmpeg -f v4l2 -framerate 10 -thread queue size 512 -i /dev/video2 -vf "rotate=0" -ss 00:00:01 -vframes 1 -map 0 stereo.png

convert -crop 50%x100% stereo.png camera.png mv camera-0.png camera2.png

mv camera-1.png camera4.png

convert -resize 320x240 camera2.png camera2.png convert -resize 320x240 camera4.png camera4.png

Then, as above, issue for example the command

./morph2quilt -l camera2.png -r camera4.png -m 9x5 -d myquilt single stereocam.png

5.3.3Quilt from two different parallel or convergent (toe-in) camera

In this case, type this long command

ffmpeg -f v4l2 -framerate 10 -video size RESOLUTION -thread queue size 512 -i /dev/video2 -vf "rotate=0"

-ss 00:00:01 -vframes 1 -map 0 camera4.png -f v4l2 -framerate 10 -video size RESOLUTION

-thread queue size 512 -i /dev/video4 -vf "rotate=0" -ss 00:00:01 -vframes 1 -map 1 camera2.png

with, e.g., input RESOLUTION equal to 240 × 320. Then, as above, issue the command

./morph2quilt -l camera2.png -r camera4.png -m 9x5 -d myquilt two cam.png

5.4Generate Native Image / Multi-Vision Stereo Vision

The MORPHOLO Library commands, that allows to convert the Quilt into a Native multi- view image –through morphing algorithms and taking into account display calibration data for specific slanted lenticular 3D monitors, are the following.

5.4.1Native image from sorted views stored in a directory

Use:

18

./images2native -q NxM QUILT -m FINAL MAP FILE DIR PATH NATIVE FILE

with

where DIR PATH is the Path to the directory where N × M sorted views (or pre-loaded images) can be found.

For example, the above command can look like this:

./images2native -q 9x5 -m mymap.map

/home/mymorpholo mynative nxm imgs.png

5.4.2Display Native image from (historical) stereo L-R images

The following command creates a Native image starting from a given set of L-R images and shows it in an external monitor via multi screen configuration:

./morpholo-display -l IMG1 -r IMG2 -m NxM QUILT -q NxM QUILT

By default, ’-screen 0’ and disparity map is used for the morphing, otherwise use the option ’-d’ to apply the DeepFlow algorthim. The option ’-s’ sets a subsampling in the morphing process to speed up calculations, e.g., use small values ’-s 2’.

For example, the above command can look like this (press ’Q’ to quit):

./morpholo-display -l photo2.png -r photo4.png -m mymap.map -q 9x5 -d -s 2 - -screen 1

5.4.3Native image output from input Quilt

To obtain the input Native image for its 3D vision in slanted lenticular displays use:

19

./morph2native -q NxM QUILT -r RESOLUTION -a FINAL MAP FILE INPUT QUILT FILE NATIVE FILE

with

For example, the above command can look like this:

./morph2native -q 9x5 -r 240x320 -a mymap.map myquilt xyz.png mynative.png

5.4.4Native image display for continuos Loop presentations

The MORPHOLO command to show a native image when multiple displays are present is:

In order to display in a Loop randomly Native images (every 12secs each and sequentially numbered as 1, · · · 14 in the directory img), the shell script below can be used:

20

#!/bin/bash

declare secs=12 declare NrImg=14 declare RMD="img/2.png"

while [ 1 ]; do

for j in $(seq 1 1 $NrImg) do

echo "Display Random Native" echo "$j $RMD"

./native2display $RMD & pid=$!

for RND in $(shuf -i 1-$NrImg -n 1) do

RMD=img/$RND.png done

echo "Changed Native ($j) to $RMD"

sleep $secs

kill SIGUSR1 $pid done

done

21

6SECTION

Streaming of 3D Videos

6.1pre-Recorded Stereoscopic Video Streaming

3D stereoscopic videos are available, e.g., in YouTube and can be freely downloaded. The processing time to convert such stereoscopic video frames into Native images in real time, can be large depending on the images size and resolution. However, it can be useful to carry out this exercise through MORPHOLO just for testing purposes.

First, prepare the archived stereoscopic video. Use the following command to rescale the video with fixed width and height (e.g., 320 × 180px):

ffmpeg -i video youtube.mp4 -vf scale=”320:180” video rescaled.mp4

To retain the aspect ratio based on the width, just give height as scale="320:-1". Second, create LUT map from your HoloPlay calibration (’.json’) file considering a small Quilt size:

./morph2native -r 128x256 -q 4x2 -m myyoutube.map mycal.json

Then, to stream the pre-recorded video issue the MORPHLO command

./morpholo-rt -c CONFIG INI -m myyoutube.map -q 4x2 -d -s 2 --screen 0

By default, Disparity map is used for the morphing, otherwise use the option ’-d’ to apply the DeepFlow algorthim. The option ’-s’ sets a subsampling in the morphing process to speed up calculations, e.g., use small values ’-s 2’.

The text CONFIG INI file (.e.g., ’config youtube.ini’) should contain:

22

[camera] devNumber=-1 width=320 height=180 fps=8

file="video rescaled.mp4" [processing]

width=256

height=128

[native]

width=2560

height=1600

6.2Elapsed Processing Time

Existing 3D TV video technologies are mainly based on the use of stereoscopic cameras to save time in the process. By using the ’mycal.json’ calibration file, and applying the light-field geometric transformations of Eq.(1) only, one in principle can also get the multi-view Native image. This simpler procedure requires considerable calculation power, since the final native image for the HoloPlay must have a resolution of 2560 × 1600px with RGB color channels -–such that, once the pixel to be mapped is fixed, the map value for each color channel implies separated calculations. In essence this procedure as such makes real-time video in 3D difficult to achieve.

However, since this mapping matrix depends on the geometric position of each pixel, and on the calibration parameters of the lenticular display, the constructed Lookup Tables (LUT) by MORPHOLO allows to reduce further the runtime computation and save processing time. As discussed above, this LUT is created only once at the beginning of the mapping process: Quilt → HoloPlay image, and then used for all the Native frames to be visualized. In Fig.15, an estimate of the total processing time employed for the generation of MOR- PHOLO holograms starting from a set of small-resolution stereoscopic images is illustrated. As quantified in these curves, total processing time steps below 0.1 sec, in between 10-to-35 different views, can be obtained via MORPHOLO with the use of LUT. This implies that in 1 second, we can generate ten or more different Native frames, opening a gateway to achieve 3D real-time video streaming.

23

Figure 15: MORPHOLO performance test on two different PC

Intel Core i3-i5 with 16G-8G RAM, respectively.

The values quantified in Fig.15 can be otaining using one of this MORPHOLO commands: morph2quilt, morph2native and morpholo-display by adding the option:

-t, --time Show elapsed processing time

For example, a shell script for estimating Quilt elapsed time in a Loop for variable number of views can be written as:

#!/bin/bash

## RUN: ./run-thiscript.sh 2> stat-morpholo-deepflow.txt

for i in $(seq 2 1 48) do

echo "Start 1 x $i ..."

## Quilt Elapsed Time x Deepflow:

./morph2quilt -l photo2.png -r photo4.png -m 1x$i -d -t myquilt stat.png

done

The output will be like this:

24

Start 1 x 44 ...

Reading files

Elapsed time (cpu time): 0.000779 s

Elapsed time (wall clock): 0.000781 s

Deepflow mode enabled

Processing step

Elapsed time (cpu time): 7.657460 s

Elapsed time (wall clock): 1.982400 s

Writing file

Elapsed time (cpu time): 0.018364 s

Elapsed time (wall clock): 0.018378 s

Start 1 x 45 ...

6.33D Real-Time Video Streaming

Create first the LUT map from your HoloPlay calibration (’.json’) file considering a small Quilt size:

./morph2native -r 128x256 -q 4x2 -m mymap.map mycal.json

To stream live 3D video issue the following command with one of the CONFIG INI text file below depending on the setup choosen:

./morpholo-rt -c CONFIG INI -m mymap.map -q 4x2 -d -s 2 --screen 0

By default, Disparity map is used for the morphing, otherwise use the option ’-d’ to apply the DeepFlow algorthim. The option ’-s’ sets a subsampling in the morphing process to speed up calculations, e.g., use small values ’-s 2’.

6.3.1Using single stereo camera with parallel lens

The CONFIG INI file (.e.g., ’config singlecam.ini’) must contain:

25

[camera]

devNumber=2

width=320

height=180

fps=8

[processing]

width=256

height=128

[native]

width=2560

height=1600

6.3.2Using two different parallel or convergent (toe-in) camera

In this case, the CONFIG INI file (.e.g., ’config twocam.ini’) must contain:

[camera]

width=320

height=180

fps=8

[camera0]

devNumber=2

[camera1]

devNumber=4

[processing]

width=256

height=128

[native]

width=2560

height=1600

26

6SECTION

References

[1]Geng J., Three-dimensional display technologies, Adv. Opt. and Photon. Vol.5 (2013) pp. 456-535. DOI:10.1364/AOP.5.000456

[2]LookUp Tables (LUT): An array that replaces runtime computation with a simpler array indexing operation. https://en.wikipedia.org/wiki/Lookup_table

[3]Baumann, R. Animating stereograms with optical flow morphing (blog), 17 Aug 2016 https://ryanfb.github.io/etc/2016/08/17/animating_stereograms_with_ optical_flow_morphing.html (accessed November 2019).

[4]EEPROM: A type of non-volatile memory used in electronic devices to store small amounts of reprogrammed data. https://en.wikipedia.org/wiki/EEPROM

[5]Olsson R. et at., Converting conventional stereo pairs to multi-view sequences using morphing, Proc. SPIE, Conf. Stereoscopic Displays and Appl. XXIII, USA, 8288 Jan (2012). DOI:10.1117/12.909253

[6]Katada K. et al., The Triangulation of Toe-in style stereo camera, Proc. 2nd Int. Conf. Intell. Sys. and Image Processing (2014). DOI:10.12792/icisip2014.007

[7]Banks, M.S., Stereoscopy and the Human Visual System, SMPTE Motion Imaging J. 121 (2012) 24. DOI:10.5594/j18173

[8]Gao, Z. et al., Correcting geometric distortions in stereoscopic 3D imaging, PLOS ONE 8 Oct. (2018). DOI:10.1371/journal.pone.0205032

[9]Stereoscopic 3D — toe-in vs parallel cameras. https://forums.cgsociety.org/t/ stereoscopic-3d-toe-in-vs-parallel-cameras/1302865

[10]Xiao J. and Shah M., Tri-view morphing, Computer Vision and Image Understanding 96 (2004) 345. DOI:10.1016/j.cviu.2004.03.014

[11]Huang H.C., Nain S-H., Cheng T., Disparity-based view morphing –a new technique for image-based rendering, VRST (1998) DOI:10.1145/293701.293703

[12]Aslam A. and Ansari S., Depth-Map Generation using Pixel Matching in Stereoscopic Pair of Images. https://arxiv.org/abs/1902.03471

[13]Weinzaepfel P. et al., DeepFlow: Large displace-ment optical flow with deep matching, ICCV-IEEE Intern. Conf. Comp. Vision, Dec (2013) Australia. DOI:10.1109/ICCV.2013.175 hal-00873592

27

[14]Seitz S.M. and Dyer C.M., View morphing, SIGGRAPH Conf. Proc., (1996) 21. DOI:10.1117/12.349368

[15]van Berkel C., Image preparation for 3D-LCD, Proc. SPIE 3639, Stereoscopic Displays and Virtual Reality Systems VI, (24 May 1999). DOI:10.1117/12.349368

[16]Takaki Y., Multiview 3D display employing a flat panel display with slanted pixel ar- rangement, Soc. Info. Display (SIF) 18 (2012) 476. DOI:10.1889/JSID18.7.476

[17]Don Lee E. et al., Upscaled sub-pixel multiplexing of multi-view images based on weighted view overlay, Electronic Lett. 51 (2015) 828. DOI:10.1049/el.2014.3668

[18]Jeong, Y.J. et al., Direct light field rendering without 2D image generation, Soc. Info. Display (SIF) 24 (2017) 686. DOI:10.1002/jsid.513

[19]Stackable, (Blog) Building and calibrating a stereo camera with OpenCV (< 50). https://erget.wordpress.com/2014/02/01/calibrating-a-stereo-camera-with-opencv

[20]Yu L. and Pan B., Single-camera stereo-digital image correlation with a four- mirror adapter: optimized design and validation, Opt. Laser Eng. (2016). DOI:10.1016/j.optlaseng.2016.03.014

[21]Gluckman J. and Nayar S.K., Rectified Catadioptric Stereo Sensors, IEEE Trans. Pat- tern Analisis and Machine Intelligence 24 (2002) 224. DOI:10.1109/34.982902

[22]Simanek D., (Blog) 3D done with mirrors and prisms: Mirror and Prism Methods for 3d Macro Photography. https://www.lockhaven.edu/˜dsimanek/3d/stereo/ 3dgallery16.htm

28