From the desk of

Jonathan Sick

  • jonathansick.ca
  • GitHub
  • RSS

Wiley et al (2011): Using MapReduce for Image Co-Addition

April 11, 2011

We report on our experience of implementing a scalable image-processing pipeline for the SDSS imaging database using Hadoop. This multiterabyte imaging data set provides a good testbed for algorithm development, since its scope and structure approximate future surveys. First, we describe MapReduce and how we adapted image co-addition to the MapReduce framework. Then we describe a number of optimizations to our basic approach and report experimental results comparing their performance.

Finally, astronomers are getting on with developing modern codes for dealing with our Big Data affliction.

  • Copyright 2010–2012 Jonathan Sick, all rights reserved.
  • Typeset in Skolar and Museo Slab by Typekit. Powered by Tumblr.
  • Elsewhere:
    • jonathansick.ca
    • GitHub
    • RSS