First about Sard: (a write-up [PDF] ). I also display a bit my hobbies: Panorama photography (since 1999, a time when panoramas were still stichted). Later with a mirror camera. Then with GoPro Max, Iphone and more recently with the insta 360 camera (I for strange tech enhousiastic reasons pride myself to have not only used almost all consumer apple products (I did not get the Newton) but also all consumer GoPro products (I have still almost all my go pros 3-12), all DJI mini drones (1,2,3,4 and now all Insta products (a go pro alternative as well as X2, X3). I recently managed to combine different hobbies of mine (Running, Drone photography and Panorama photography), by strapping the X3 onto the Avata or run with the Avata, or run with the 360 degree camera (see for example my life in a day youtube videos from 2010 and from 2020). The 2010 one was done with my Mirror camera, the 2020 one with a GoPro Max. But these are more than just hobbies. I like technology also for their connections with mathematics (obvious for photography, computers or drones) but also for running, as I do almost all my “deep thinking” (generating ideas) while running, which is beneficial not only because of the fresh air but also because one is not strapped into any electronic device which can be very distracting). I worked with Jose Ramirez-Herran in 2007 on a structure of motion problems. We had pretty cool results, like improvements on the classical Theorem of Ullman in Computer vision with explicit implementations of the inverse map (a nonlinear problem for 3 affine cameras observing 3 points), improving that classical result of Ullman. Then we solved the inverse problem for omni-directional cameras and identified when a reconstruction is possible. We also formulated a basic structure from motion inequality which just counts dimensions but gives an idea how many cameras and how many points are needed in order to reconstruct both camera and point locations from the photo plates. To the definition of a “camera” (I change a bit notation to the paper we wrote then). A camera is a map Q from a manifold M to where all are equivalent to a manifold N, such that . The camera comes with a parameter manifold S. There can also be a global symmetry group G acting on both M and so N. Now, the dimension d of the space manifold M, the dimension k of the retina manifold N, the dimension s of the camera manifold, and the dimension g of the group allow to determine whether n cameras (n photos) and m points allow from the n*m data points to reconstruct both the points and the cameras. By counting dimensions, we get thestructure of motion inequality. This is a place where sub-manifolds of manifolds appear. The connection to discrete mathematics is obvious. If we look at what happens really in a digital camera is that we have a discrete set of points which matter in all aspects (the resolution of the camera determines that). An example is when the cameras are orthographic cameras where one has Ullman’s theorem. Here is an explicit implementation of the inverse map in Mathematica: we have m= 3 points and n=3 cameras. Space M is the 2-dimensional plane so that d=2. The submanifolds N are lines of dimension k=1. The dimension of the parameter space of the cameras is 1 as it is the real projective line (a circle). There is a global isometry symmetry group, the nonlinear translation rotation group (affine group) which has dimension 3 (2 translation dimensions and 1 rotation dimension). The indquality tells . It is satisfied for n=3 and m=3 which shows that (at least locally) we have a unique reconstruction. The situation is very similar for 2D omnidirectional cameras, where the dimensions are the same. Here are some slides from 2007, when talking at Colby on October 29, 2007[PDF]. There was an earlier talk at the Harvard Summer school (when Andy Engelward had been putting together nice summer talks for a general audience). I explain there for a general audience how 3 points and 3 cameras produce 9 angles. This does not seem to get back the 2*6 coordinates of the cameras and points, but there is also a scale (conformal) and translational invariance which gives a symmetry group of dimension 3. Again, 3 points and 3 cameras should allow to reconstruct the points and cameras. The omni-directional situation is actually simpler in that we get a linear system of equations of 9 angles and 9 coordinates. I wrote for the talk down the matrix of the equation and computed the determinant.Then I made the point that sometimes algebra is less powerful than geometry (something Otto Bretscher, who had invited me to Colby and who has written a famous textbook in linear algebra did not like to hear, as he told me afterwards, he did not like linear algebra being attacked …). In any case, our Theorem (Knill-Ramirez) tells that if we have 3 omni-cameras whtat a noncollinear and 3 points that a non-collinear and the union of cameras and points is not contained in the union of two lines, then the structure of motion problem for omnidirectional 2D cameras has a unique solution. Examples show that this is sharp. We have identified all points where the determinant of the structure of motion map on the 12 dimensional space is invertible. The slides then continue and look at the Ullman theorem from 1979. This is some mathematics which comes in when I fly a 360 degree camera (Insta X3) with my FPV drone (DJI avata) in my office, the common room at the math department (at the end) and in the Cambridge commons. I stayed cautions because I do would not have the funds at the moment to replace the drone or the camera. I currently do not know where the limits are. But the drone balances an unstable equilibrium and we know from catastrophe theory, that singularities in the parameter space can easily be reached. There is no question that like in the boat tipping example, the “drone tipping” would happen catastrophically and that the rone would not recover once the stick is inverted. I damaged a DJI mini (1) and a GoPro Max which I had strapped a few years ago onto the drone. It took off from the balcony of the math department but then tumbled down to the fountain below the science center damaging both.
