MaiZaic: a robust end-to-end pipeline for mosaicking freely flown aerial video of agricultural fields

Unmanned aerial vehicles (UAVs) are increasingly used for high throughput phenotyping. In principle, freely flown vehicles would permit real-time flexibility in identifying and scouting regions of interest. Mosaicking multiple images provides a high resolution global image and consumer-grade UAVs offer low cost, ease of flying, and excellent RGB cameras. The vehicles inaccurate telemetry complicates estimating the homographies between pairs of frames, the standard mosaicking approach. Moreover, errors accumulate during computation, distorting later portions of the mosaic. Finally, crop fields are particularly challenging to mosaic because their planting is so regular and the plants are so similar, eliminating distinctive features that could guide mosaicking. We propose MaiZaic, an end-to-end pipeline that dynamically samples video frames using optical flow, automates camera and gimbal calibration, estimates homographies with an unsupervised convolutional neural network, detects shots among frames, and generates mini-mosaics. Together, these techniques significantly reduce errors in the output mosaics. Our deep learning model is trained on a comprehensive video dataset comprising different flight trajectories, maize lines, growth stages, and augmented illumination data. MaiZaic is more accurate and faster than ASIFT and more robust than our earlier CorNet and CorNetv2. We demonstrate MaiZaics effectiveness in generating accurate mosaics of imagery captured by freely-flown UAVs and explore its generalizability. Core ideasO_LIMaiZaic is an end-to-end pipeline to mosaic freely flown agricultural imagery captured with consumer-grade UAVs. C_LIO_LIMaiZaic introduces novel algorithms that efficiently choose video frames, calibrate, and mosaic the imagery. C_LIO_LIOur unsupervised deep homography estimator, CorNetv3, is 14 times faster and 8.59% more accurare than ASIFT. C_LIO_LIMaiZaic generalizes well and mosaicks maize at different growth stages, objects, trajectories, cameras, and pilots. C_LIO_LIThe mini-mosaicking algorithm improves mosaic accuracy by interrupting error accumulation. C_LI