Calibrating Camera Parameters

Problem Description

Cameras are able to capture 2D images of 3D objects. The Ray Tracer Camera object tries to simulate the behavior of a real world camera, by parameterizing the position, focal length, field of view, etc. In an ideal scenario we would try to reconstruct all these parameters of the camera simultaneously. However for the sake of this demo we shall try to predict the focus of the camera and the position of the camera which allows us to generate an image similar to the target image.

Setup

Let us get started by importing the necessary packages. RayTracer implements the necessary rendering functionalities and depends on Zygote for computing its derivatives. Apart from these two main packages we need Images and Plots for displaying the rendered images and finally Flux for its Optimizers.

NOTE: Any other library like Optim may also be used for optimization.

pkg"add https://github.com/avik-pal/RayTracer.jl"
pkg"add Images Flux#sf/zygote_updated Zygote#master IRTools#master"
pkg"precompile"
using RayTracer, Statistics, Zygote, Flux, Images, Plots

305.1s

Julia

Scene Rendering in RayTracer

In this section we shall setup the parameters of the scene. This has been discussed in details in a prior tutorial. If you haven't read that, do so before proceeding any furthur.

screen_size = (w = 400, h = 300)

light = PointLight(Vec3(1.0), 100000.0, Vec3(0.0, 0.0, -10.0))

0.5s

Julia

Point Light Color - x = 1.0, y = 1.0, z = 1.0 Intensity - 100000.0 Position - x = 0.0, y = 0.0, z = -10.0

We shall have only one object in our scene for this demonstration. The object will be a rectangle on the z = 0 plane. We manually triangulate the rectangle and create a scene. However, RayTracer provides a convenience function triangulate_faces which can do this thing for any polygon mesh.

scene = [
    Triangle(Vec3(20.0, 10.0, 0.0), Vec3(20.0, -10.0, 0.0),
             Vec3(-20.0, 10.0, 0.0), color = rgb(0.0, 1.0, 0.0)),
    Triangle(Vec3(-20.0, -10.0, 0.0), Vec3(20.0, -10.0, 0.0),
             Vec3(-20.0, 10.0, 0.0), color = rgb(0.0, 1.0, 0.0))
]

0.3s

Julia

2-element Array{Triangle{Array{Float64,1}},1}: Triangle Object: Vertex 1 - x = 20.0, y = 10.0, z = 0.0 Vertex 2 - x = 20.0, y = -10.0, z = 0.0 Vertex 3 - x = -20.0, y = 10.0, z = 0.0 Material: Plain Color - (x = 0.0, y = 1.0, z = 0.0), Reflection - 0.5 Triangle Object: Vertex 1 - x = -20.0, y = -10.0, z = 0.0 Vertex 2 - x = 20.0, y = -10.0, z = 0.0 Vertex 3 - x = -20.0, y = 10.0, z = 0.0 Material: Plain Color - (x = 0.0, y = 1.0, z = 0.0), Reflection - 0.5

Next we generate our target image.

cam = Camera(Vec3(0.0, 0.0, -30.0), Vec3(0.0, 0.0, 0.0), Vec3(0.0, 1.0, 0.0),
             90.0, 1.0, screen_size.w, screen_size.h)

origin, direction = get_primary_rays(cam)

color_target = raytrace(origin, direction, scene, light, origin, 2)

plot(get_image(color_target, screen_size.w, screen_size.h))

1.4s

Julia

Optimizing the Camera Parameters

Now let us discuss how to recover the actual focus of the camera. First we shall produce an initial guess of our focus. Now we generate an image of the rectangle we produced above. Note that this object can be anything, as long as we have an image of it using the original focus and position. Next we compute the L1 Norm of the difference between the 2 images. Using this value we can compute the derivative wrt the camera parameters. Then we can use any first order optimizer to optimize the parameters.

Starting with an initial guess of the parameters

cam_guess = Camera(Vec3(5.0, -4.0, -20.0), Vec3(0.0, 0.0, 0.0),
                   Vec3(0.0, 1.0, 0.0), 90.0, 3.0, screen_size.w, 
                   screen_size.h)

origin_guess, direction_guess = get_primary_rays(cam_guess)

color_guess = raytrace(origin_guess, direction_guess, scene, light,
                       origin_guess, 2)

plot(get_image(color_guess, screen_size.w, screen_size.h))

2.2s

Julia

Let us plot the target and the initial images side by side to get a rough idea of what we are trying to do.

plot_images(im1, im2, title = "Initial Guess") =
    plot(plot(im1, title = "Target Image"),
         plot(im2, title = title),
         plot(map(x -> RGB(abs(x.r), abs(x.g), abs(x.b)), im1 - im2),
              title = "Difference"))

1.0s

Julia

plot_images (generic function with 2 methods)

im1 = get_image(color_target, screen_size.w, screen_size.h)
im2 = get_image(color_guess, screen_size.w, screen_size.h)
plot_images(im1, im2)

1.4s

Julia

Processing the Ray Tracer output

The output of the raytrace function is a Vec3 object. In order to perform operations on it we should transform it back to the normal array format. Though most of these operations can be performed on Vec3 objects it is highly recommended not to do so.

function process_image(im, width, height)
    color_r = reshape(im.x, width, height)
    color_g = reshape(im.y, width, height)
    color_b = reshape(im.z, width, height)

    im_arr = reshape(hcat(color_r, color_g, color_b), (width, height, 3))
    
    return zeroonenorm(im_arr)
end

0.8s

Julia

process_image (generic function with 1 method)

img = process_image(color_target, screen_size...);

0.5s

Julia

Optimization using First Order Optimizer

Finally we shall define the optimizer and iteratively optimize the camera focus.

opt = ADAM(1.0)

1.3s

Julia

ADAM(1.0, (0.9, 0.999), IdDict{Any,Any}())

for i in 1:50
    gs = Zygote.gradient(Params([cam_guess])) do
        origin_guess, direction_guess = get_primary_rays(cam_guess)
        color_guess = raytrace(origin_guess, direction_guess, scene,
                               light, origin_guess, 2)
        img_guess = process_image(color_guess, screen_size.w, screen_size.h)
        loss = sum(abs.(img .- img_guess))
        @show loss
        return loss
    end
    update!(opt, cam_guess.focus, gs[cam_guess].focus)
    update!(opt, cam_guess.lookfrom, gs[cam_guess].lookfrom)
    if i % 10 == 0
        @info "$i iterations completed"
        origin_guess, direction_guess = get_primary_rays(cam_guess)
        color_guess = raytrace(origin_guess, direction_guess, scene,
                               light, origin_guess, 2)
        img_guess = get_image(color_guess, screen_size.w, screen_size.h)
        @show cam_guess
    end
end

439.4s

Julia

Let us plot the new image generated with the calibrated camera.

origin_guess, direction_guess = get_primary_rays(cam_guess)
color_guess = raytrace(origin_guess, direction_guess, scene,
                       light, origin_guess, 2)
img_guess = get_image(color_guess, screen_size.w, screen_size.h)
plot_images(im1, img_guess, "Image with Calibrated Camera")

1.6s

Julia

Conclusion

As you can see from the above plot we have been able to nearly reconstruct the original image. One interesting thing to notice is that out optimized parameters don't exact match with the original parameters, especially the focus and the z-coordinate of the lookfrom vector. This is because the two values are highly correlated and moving the camera forward is essentially same as zooming into the image.