Recent Patents in Signal Processing (November 2017) – Image and audio inpainting

For our November 2017 issue, we cover recent patents granted in the area of image and audio inpainting

Patent no. 9,583,111 presents a method for packet loss concealment, that includes: continuously receiving a digital audio stream; extracting audio features from the digital audio stream while the digital audio stream is unharmed; and upon detecting a gap in the digital audio stream, filling the gap with one or more previous segments of the digital audio stream, wherein the filling is based on a matching of the one or more of the extracted audio features with one or more audio features adjacent to the gap.

The invention no. 9,443,285 introduces a method for replacing image data in a destination region that is divided into sub-pieces along one or more cutting paths, which start and end at two different points on the border, and finding replacement data for the sub-pieces. The cutting paths may be determined as a function of the type of image structured at the start and the end points. The cutting paths may also be determined as a function of the area of the sub-pieces and the lengths of the cutting paths. Optionally, the destination region may be determined by a spot detection algorithm. Further optionally, the spot detection algorithm may comprise calculation of a high pass filter, or detection of areas of luminosity and border-to-volume ratios. A method for moving an image element within an image is also provided.

In patent no. 9,378,583 an apparatus and method for bidirectionally inpainting an occlusion area appearing during generation of a virtual viewpoint image, using a foreground area and a background area based on a predicted volume are provided. The method includes receiving an input of a depth image and a color image, the depth image having depth information for a first viewpoint, and the color image having color information for the first viewpoint, generating a virtual viewpoint image at a second viewpoint based on the depth image and the color image, separating the virtual viewpoint image into a foreground area and a background area, based on the depth information and direction information regarding a direction from the first viewpoint to the second viewpoint, predicting a three-dimensional (3D) volume of the foreground area, and inpainting an occlusion area bidirectionally using the foreground area and the background area, based on the predicted 3D volume, the occlusion area being included in the virtual viewpoint image.

In patent no. 9,256,926 a method of processing an image signal comprising image and depth information is provided. The method is configured to perform segmentation on an image based on depth/disparity information present in the image signal comprising said image, and subsequently inpaint background for correction of the errors in the image around the foreground objects into a region that extends beyond the segment boundary of the foreground object and/or inpaint foreground for correction of errors in the image into a region that extends inside the segment boundary of the foreground object. In this way compression and other artifacts may be reduced.

As descibed in patent no. 9,230,309, the image processing apparatus inpaints a part of an image displayed on the display unit. The control unit determines a removal patch including a removal region and a first non-removal region that is a region that does not include the removal region in the image, and replace pixel values of pixels included in the removal region with pixel values of pixels outside the removal patch. The control unit calculates a distance from the removal region for pixels included in the first non-removal region, blends pixel values of the pixels included in at least a portion of the first non-removal region with the pixel values of the pixels outside the removal patch based on the calculated distance to obtain blended pixel values, and replaces the pixel values of the pixels included in the at least a portion of the first non-removal region with the blended pixel values.

In patent no. 9,224,052 a method for in-image periodic noise pixel inpainting is provided. It is determined whether a current frame includes periodic noise pixels, and locations of periodic noise pixels are identified. Non-periodic-noise pixels in a reference frame are utilized to inpaint the periodic noise pixels in the current frame.

Following patent no. 9,165,347, an apparatus, system, method, and article to continue border lines into an unknown region of an image from a known background; determine segments, based on the continued borders, for the unknown region of the image; and propagate pixels from a known area of the image to the unknown area based on the determined segments and continued borders.

In patent no. 9,106,892 systems and methods provide image compression based on parameter-assisted inpainting. In one implementation of an encoder, an image is partitioned into blocks and the blocks classified as smooth or unsmooth, based on the degree of visual edge content and chromatic variation in each block. Image content of the unsmooth blocks is compressed, while image content of the smooth blocks is summarized by parameters, but not compressed. The parameters, once obtained, may also be compressed. At a decoder, the compressed image content of the unsmooth blocks and the compressed parameters of the smooth blocks are each decompressed. Each smooth block is then reconstructed by inpainting, guided by the parameters in order to impart visual detail from the original image that cannot be implied from the image content of neighboring blocks that have been decoded.

If you have an interesting patent to share when we next feature patents related to image and audio inpainting, or if you are especially interested in a signal processing research field that you would want to be highlighted in this section, please send email to Csaba Benedek (benedek.csaba AT sztaki DOT mta DOT hu).

References

Number: 9,583,111
Title: Example-based audio inpainting
Inventors: Schechner; Yoav (Kiryat Bialik, IL), Elad; Michael (Kiriat Tivon, IL), Bahat; Yuval (Tel Aviv, IL)
Issued: February 28, 2017
Assignee: Technion Research & Development Foundation Ltd. (Haifa, IL)

Number: 9,443,285
Title: Method for sliced inpainting
Inventors: Kokemohr; Nils (Hamburg, DE)
Issued: September 13, 2016
Assignee: Google Inc. (Mountain View, CA)

Number: 9,378,583
Title: Apparatus and method for bidirectionally inpainting occlusion area based on predicted volume
Inventors: Lee; Seung Kyu (Seoul, KR), Lim; Hwa Sup (Hwaseong-si, KR), Lee; Kee Chang (Yongin-si, KR), Kim; Do Kyoon (Seongnam-si, KR)
Issued: June 28, 2016
Assignee: Samsung Electronics Co., Ltd. (Gyeonggi-do, KR)

Number: 9,256,926
Title: Use of inpainting techniques for image correction
Inventors: Berretty; Robert-Paul Mario (Eindhoven, NL), Magalhaes; Jose Pedro (Utrecht, NL), Klein Gunnewiek; Reinier Bernardus Maria (Eindhoven, NL), Barenbrug; Bart Gerard Bernard (Eindhoven, NL), Shao; Ling (Eindhoven, NL)
Issued: February 9, 2016
Assignee: Koninklijke Philips N.V. (Eindhoven, NL)

Number: 9,230,309
Title: Image processing apparatus and image processing method with image inpainting
Inventors: Nagaoka; Eiichi (Hyogo, JP)
Issued: January 5, 2016
Assignee: Panasonic Intellectual Property Management Co., Ltd. (Osaka, JP)

Number: 9,224,052
Title: Method for in-image periodic noise pixel inpainting
Inventors: Lin; Che-Tsung (Hsinchu, TW), Lin; Yu-Chen (Taipei, TW)
Issued: December 29, 2015
Assignee: Industrial Technology Research Institute (Hsinchu, TW)

Number: 9,165,347
Title: Method of and apparatus for local optimization texture synthesis 3-D inpainting
Inventors: Yurkov; Kirill Valerjevich (Saint-Petersburg, RU), Belyakov; Alexei Vladimirovich (Saint-Petersburg, RU)
Issued: October 20, 2015
Assignee: Intel Corporation (Santa Clara, CA)

Number: 9,106,892
Title: Image compression based on parameter-assisted inpainting
Inventors: Sun; Xiaoyan (Beijing, CN), Xiong; Zhiwei (Beijing, CN), Wu; Feng (Beijing, CN), Li; Shipeng (Palo Alto, CA)
Issued: August 11, 2015
Assignee: Microsoft Technology Licensing, LLC (Redmond, WA)