# Calibrating Camera Parameters

## Problem Description

Cameras are able to capture 2D images of 3D objects. The Ray Tracer Camera object tries to simulate the behavior of a real world camera, by parameterizing the position, focal length, field of view, etc. In an ideal scenario we would try to reconstruct all these parameters of the camera simultaneously. However for the sake of this demo we shall try to predict the focus of the camera and the position of the camera which allows us to generate an image similar to the target image.

## Setup

Let us get started by importing the necessary packages. RayTracer implements the necessary rendering functionalities and depends on Zygote for computing its derivatives. Apart from these two main packages we need Images and Plots for displaying the rendered images and finally Flux for its Optimizers.

NOTE: Any other library like Optim may also be used for optimization.

```pkg"add https://github.com/avik-pal/RayTracer.jl"
pkg"precompile"
using RayTracer, Statistics, Zygote, Flux, Images, Plots```

## Scene Rendering in RayTracer

In this section we shall setup the parameters of the scene. This has been discussed in details in a prior tutorial. If you haven't read that, do so before proceeding any furthur.

```screen_size = (w = 400, h = 300)

light = PointLight(Vec3(1.0), 100000.0, Vec3(0.0, 0.0, -10.0))```
Point Light Color - x = 1.0, y = 1.0, z = 1.0 Intensity - 100000.0 Position - x = 0.0, y = 0.0, z = -10.0

We shall have only one object in our scene for this demonstration. The object will be a rectangle on the z = 0 plane. We manually triangulate the rectangle and create a scene. However, RayTracer provides a convenience function triangulate_faces which can do this thing for any polygon mesh.

```scene = [
Triangle(Vec3(20.0, 10.0, 0.0), Vec3(20.0, -10.0, 0.0),
Vec3(-20.0, 10.0, 0.0), color = rgb(0.0, 1.0, 0.0)),
Triangle(Vec3(-20.0, -10.0, 0.0), Vec3(20.0, -10.0, 0.0),
Vec3(-20.0, 10.0, 0.0), color = rgb(0.0, 1.0, 0.0))
]```
2-element Array{Triangle{Array{Float64,1}},1}: Triangle Object: Vertex 1 - x = 20.0, y = 10.0, z = 0.0 Vertex 2 - x = 20.0, y = -10.0, z = 0.0 Vertex 3 - x = -20.0, y = 10.0, z = 0.0 Material: Plain Color - (x = 0.0, y = 1.0, z = 0.0), Reflection - 0.5 Triangle Object: Vertex 1 - x = -20.0, y = -10.0, z = 0.0 Vertex 2 - x = 20.0, y = -10.0, z = 0.0 Vertex 3 - x = -20.0, y = 10.0, z = 0.0 Material: Plain Color - (x = 0.0, y = 1.0, z = 0.0), Reflection - 0.5

Next we generate our target image.

```cam = Camera(Vec3(0.0, 0.0, -30.0), Vec3(0.0, 0.0, 0.0), Vec3(0.0, 1.0, 0.0),
90.0, 1.0, screen_size.w, screen_size.h)

origin, direction = get_primary_rays(cam)

color_target = raytrace(origin, direction, scene, light, origin, 2)

plot(get_image(color_target, screen_size.w, screen_size.h))```

## Optimizing the Camera Parameters

Now let us discuss how to recover the actual focus of the camera. First we shall produce an initial guess of our focus. Now we generate an image of the rectangle we produced above. Note that this object can be anything, as long as we have an image of it using the original focus and position. Next we compute the L1 Norm of the difference between the 2 images. Using this value we can compute the derivative wrt the camera parameters. Then we can use any first order optimizer to optimize the parameters.

### Starting with an initial guess of the parameters

```cam_guess = Camera(Vec3(5.0, -4.0, -20.0), Vec3(0.0, 0.0, 0.0),
Vec3(0.0, 1.0, 0.0), 90.0, 3.0, screen_size.w,
screen_size.h)

origin_guess, direction_guess = get_primary_rays(cam_guess)

color_guess = raytrace(origin_guess, direction_guess, scene, light,
origin_guess, 2)

plot(get_image(color_guess, screen_size.w, screen_size.h))```

Let us plot the target and the initial images side by side to get a rough idea of what we are trying to do.

```plot_images(im1, im2, title = "Initial Guess") =
plot(plot(im1, title = "Target Image"),
plot(im2, title = title),
plot(map(x -> RGB(abs(x.r), abs(x.g), abs(x.b)), im1 - im2),
title = "Difference"))```
plot_images (generic function with 2 methods)
```im1 = get_image(color_target, screen_size.w, screen_size.h)
im2 = get_image(color_guess, screen_size.w, screen_size.h)
plot_images(im1, im2)```

### Processing the Ray Tracer output

The output of the raytrace function is a Vec3 object. In order to perform operations on it we should transform it back to the normal array format. Though most of these operations can be performed on Vec3 objects it is highly recommended not to do so.

```function process_image(im, width, height)
color_r = reshape(im.x, width, height)
color_g = reshape(im.y, width, height)
color_b = reshape(im.z, width, height)

im_arr = reshape(hcat(color_r, color_g, color_b), (width, height, 3))

return zeroonenorm(im_arr)
end```
process_image (generic function with 1 method)
`img = process_image(color_target, screen_size...);`

### Optimization using First Order Optimizer

Finally we shall define the optimizer and iteratively optimize the camera focus.

`opt = ADAM(1.0)`
```for i in 1:50
origin_guess, direction_guess = get_primary_rays(cam_guess)
color_guess = raytrace(origin_guess, direction_guess, scene,
light, origin_guess, 2)
img_guess = process_image(color_guess, screen_size.w, screen_size.h)
loss = sum(abs.(img .- img_guess))
@show loss
return loss
end
update!(opt, cam_guess.focus, gs[cam_guess].focus)
update!(opt, cam_guess.lookfrom, gs[cam_guess].lookfrom)
if i % 10 == 0
@info "\$i iterations completed"
origin_guess, direction_guess = get_primary_rays(cam_guess)
color_guess = raytrace(origin_guess, direction_guess, scene,
light, origin_guess, 2)
img_guess = get_image(color_guess, screen_size.w, screen_size.h)
@show cam_guess
end
end```

Let us plot the new image generated with the calibrated camera.

```origin_guess, direction_guess = get_primary_rays(cam_guess)
color_guess = raytrace(origin_guess, direction_guess, scene,
light, origin_guess, 2)
img_guess = get_image(color_guess, screen_size.w, screen_size.h)
plot_images(im1, img_guess, "Image with Calibrated Camera")```

## Conclusion

As you can see from the above plot we have been able to nearly reconstruct the original image. One interesting thing to notice is that out optimized parameters don't exact match with the original parameters, especially the focus and the z-coordinate of the lookfrom vector. This is because the two values are highly correlated and moving the camera forward is essentially same as zooming into the image.