extensions package

Submodules

extensions.augmenter module

Author: Benjamin Anderson Garrard

This script houses functions that transform images and the bounding boxes associated with them.

class cvjson.extensions.augmenter.Augmenter(cvj)

Bases: cvjson.cvj.CVJ

The Augment class is used to augment images using the imgaug library. just pip install imgaug if you do not have it.

augment_geo(patch, bboxes)

This method uses the imaug library https://github.com/aleju/imgaug to implement all augmentations since it can handle bounding boxes also.

Rotation seems to rescale the bounding boxes too much so the rotation and flips should be handled by Detectron and not by this method

Parameters:
  • patch (numpy array) – This numpy array is the image. Opencv, skimage, or numpy can generate this.
  • bboxes (numpy array) – This numpy array holds the bounding boxes that are associated with this image. They are needed to be in the format of [[x1, y1, x2, y2]
Returns:

  • numpy array (The first parameter “img_aug” returned is the numpy array for the image.)
  • numpy array (The second parameter “bbox_” returned is the numpy array for the bounding boxes that have been) – been transformed with the image returned in the first parameter.
  • numpy array (The third parameter “patch” is the numpy array for the image before it was altered.)
  • numpy array (The fourth parameter “previous_boxes” returned is the numpy array for the bounding boxes that have) – not been altered associated with the returned third parameter.
fancy_pca(img, alpha_std=0.1)

Fancy PCA is a photometric data augmentation technique that finds the most important color values, 3 because of the 3 channels, then gets a scaled intensity of the colors and augments with those intensity * a random variable.

Parameters:
  • img (numpy array with (h, w, rgb) shape, as ints between 0-255)) –
  • alpha_std (how much to perturb/scale the eigen vecs and vals the paper used std=0.1) – (Default value = 0.1)
Returns:

  • numpy array (image-like array as float range(0, 1))
  • NOTE (Depending on what is originating the image data and what is receiving)
  • the image data returning the values in the expected form is very important
  • in having this work correctly. If you receive the image values as UINT 0-255
  • then it’s probably best to return in the same format. (As this
  • implementation does). If the image comes in as float values ranging from
  • 0.0 to 1.0 then this function should be modified to return the same.
  • Otherwise this can lead to very frustrating and difficult to troubleshoot
  • problems in the image processing pipeline.
  • This is ‘Fancy PCA’ from
  • # http (//papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)
  • #######################
  • #### FROM THE PAPER ###
  • #######################
  • ”The second form of data augmentation consists of altering the intensities
  • of the RGB channels in training images. Specifically, we perform PCA on the
  • set of RGB pixel values throughout the ImageNet training set. To each
  • training image, we add multiples of the found principal components, with
  • magnitudes proportional to the corresponding eigenvalues times a random
  • variable drawn from a Gaussian with mean zero and standard deviation 0.1.
  • Therefore to each RGB image pixel Ixy = [I_R_xy, I_G_xy, I_B_xy].T
  • we add the following quantity
  • [p1, p2, p3][α1λ1, α2λ2, α3λ3].T
  • Where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance
  • matrix of RGB pixel values, respectively, and αi is the aforementioned
  • random variable. Each αi is drawn only once for all the pixels of a
  • particular training image until that image is used for training again, at
  • which point it is re-drawn. This scheme approximately captures an important
  • property of natural images, namely, that object identity is invariant to
  • change.”
  • ### END ###############
  • Other useful resources for getting this working
  • # https (//groups.google.com/forum/#!topic/lasagne-users/meCDNeA9Ud4)
  • # https (//gist.github.com/akemisetti/ecf156af292cd2a0e4eb330757f415d2)
transf_one_patch(patch, bboxes, PCA_Augment=True, edge_enhancement=False, color_jittering=True, preview=False, preview_boxes=False)

This method is used to transform just one image at a time, this method also transforms the bounding boxes accordingly.

Rotation and flipping is not handled within this function. #TODO: create a method that does the rotation and flipping. The reason for this is that the bounding boxes that are transformed become too large for small object detection.

https://arxiv.org/pdf/1708.06020.pdf is a paper explaining why these augmentations were chosen.

Parameters:
  • patch (numpy array) – The image to augment
  • bboxes (numpy array) – A numpy array of the bounding boxes associated with the supplied image/patch
  • PCA_Augment (bool) – (Default value = True) Is a boolean that augments the image based on PCA color augmentation. PCA color augmentation just means taking scaled intensities of the most import colors and multiplying them through the image.
  • edge_enhancement (bool) – (Default value = False) Is a boolean that starts a convolution that enhances the edges in the image and makes them more prominent. This could make classifier find edges more quickly and then find more complex features associated with the edges in later convolutions if using deep learning.
  • color_jittering (bool) – (Default value = True) This is a boolean to start adding pixel values ranging from -20 to 20 to RGB channels randomly
  • preview (bool) – (Default value = False) Is a boolean that starts up a comparison from the image/patch supplied to the new augmented one.
  • preview_boxes (bool) – (Default value = False) Is a boolean that allows the boxes to be drawn on the images in the preview. The original and transformed versions.
Returns:

  • numpy array (The first numpy array is the image/patch)
  • numpy array (The second numpy is the bounding boxes associated with the image.)

extensions.cropper module

Primary Author: Benjamin Garrard’

class cvjson.extensions.cropper.Cropper(cvj, image_folder_path=None)

Bases: cvjson.cvj.CVJ

This class is an extension to the CVJ object. This classes sole purpose is to crop images and produce new images. This class uses the Augmenter class to generate new augmentations on the images produced if wanted. #TODO replace the augment stuff with the augmenter

crop_images_bbox_centered(save_directory, max_image_size=768, min_image_size=128, max_bounding_box_size=600, scale=1, padding=0, padded_beforehand=False, image_threshold=0, cv2enum=2, testing=0)

This method takes every image found in the image_folder_path that was supplied and then gets every bounding box that is associated with each image and then crops an image to the size specified to this method, with each bounding box being centered in at leas one image. So this means that if you have 600 bounding boxes you will get 600 image crops with each crop having a bounding box in the center. Now there can be more since this also will want to augment images. Just specify the image threshold and thats how many images each class will have where a bounding box of that class is centered inside the JSON.

Files produced:

1: {TIMESTAMP}_coco_train.json

DESCRIPTION : The file that is produced for training. If the image threshold

was supplied then this file will be stratified and have undersampled the images more strongly associated with the class that is most dominant in the json. It also oversamples the rare classes through augmentation using the imgaug library

Format = COCO

2: {TIMESTAMP}_stratified_images.json

DESCRIPTION : The file that is produced during strafitication. It has the filepaths

for all the images that are stored in {TIMESTAM}_coco_train.json

Format = None

Keys = Class Ids

values = all images associated with each class including augments
up to the threshold

3: {TIMESTAMP}_image_class_counts.json

DESCRIPTION : This file actually has the filepaths to each image associated with a class.

This is used for counts in the program and is why it is named like that.

Format = None

keys = class_id

values = filepaths to all of the images associated with each class including augments
regardless of the threshold

4: {TIMESTAMP}_augmented_image_class_filepaths.json

DESCRIPTION : The file that is produced when the stratification occurs and gives the

filepaths to the augmented images. The path’s will break if moved so it is up to the user how they want to handle this file.

Format = None

Keys = class ids

values = filepaths to augmented images associated to each class

5: {TIMESTAMP}_too_big.json

DESCRIPTION : This file contains the images and annotations for any bounding box size that was over the

maximum bounding box size set.

Format = COCO

Parameters:
  • save_directory (string) – The path to the directory where this will be saved.
  • max_image_size (int, if a float is supplied then it will be truncated.) – (Default value = 768) The maximum size an image can be. If the min_image_size is set to the same as this then all images will be square of the size supplied. Otherwise there will be a range between the two.
  • min_image_size (int, if a float is supplied then it will be truncated.) – (Default value = 128) The minimum size an image can be. If the min_image_size is set to the same as this then all images will be square of the size supplied. Otherwise there will be a range between the two.
max_bounding_box_size : int, if a float is supplied then it will be truncated.
(Default value = 600) The maximum size a bounding box can be.
scale : int, if a float is supplied then it will be truncated.
(Default value = 1) This is used to give take to scale get an image of the size of the bounding box scaled and squared. It is not exact scaling but pretty close. The min and max sizes will allow the images to be within range of those, but still give some dynamics to the images being cropped if the scale is above 1. However if 1 is used, it is by default, there will still be some dynamics in crops unless the min and max image sizes are set to the same thing.
padding : int, if a float is supplied then it will be truncated.
(Default value = 0) This parameter is used to pad images given the opencv padding enum supplied. If the padding is zero, it is by defualt, then the cropping will assume that your images were padded previously and not try to do it again. If you receive an index out of bounds error, then most likely a bounding box was near and edge and there wasn’t enough image left to do the crop and increasing this parameter should clear that up.
padded_beforehand : bool
(Default value = False) This tell the method if the image was padded beforehand. If this is false then the method will pad image supplied by the amount of padding that is supplied. However, the bounding boxes will not be tracked. The best use for this is just to use the Painter class to pad, track, and inpaint and then use the cropper and keep this True and the padding match the padding used in the Painter class.
image_threshold : int, if a float is supplied then it will be truncated.
(Default value = 0) This parameter is what makes sure that the amount of images associated to each class is the same. This is what stratifies your images in the JSON and if there aren’t enough images original to meet the threshold then more will be generated using the Augment class. If the threshold is 0, it is by default, then cropping will be unlimited and no stratification will occur.
cv2enum : int, however this is the same as the opencv border types.
(Default value = cv2.BORDER_REFLECT) This parameter is the same as opencv’s border parameters. * BORDER_REPLICATE: aaaaaa|abcdefgh|hhhhhhh * BORDER_REFLECT: fedcba|abcdefgh|hgfedcb * BORDER_REFLECT_101: gfedcb|abcdefgh|gfedcba * BORDER_WRAP: cdefgh|abcdefgh|abcdefg * BORDER_CONSTANT: iiiiii|abcdefgh|iiiiiii with some specified ‘i’
testing : int
This int is used for unit tests since this method is all about creating crops, writing them out and generating files. We don’t need to test over a whole gigantic json all the time as well as images. Also we don’t want to have to create a new testing json. So setting this int to anything other than 0 makes the method only iterate through that many images. So if it is set to 2 then it will generate crops from 2 images only.
Returns:Timestamp of when the method was called.
Return type:str

Example

If I have 7000 bounding boxes 3 classes where class 1 has 4000 bounding boxes, class 2 has 2000, and class 3 has 1000. If the image threshold is set to 3000 then class 1 will not have any augmentation and also only 3000 of the images associated with class one will be included in the {TIMESTAMP}_stratified_images_train.json that is produced. Class 2 will have 1k of augments and class 3 will have 2k of augments.

These augments rotate through the original images until the limit is reached and uses the Augment class to augment the images.

NOTE: No matter what this method will always generate as many images as there are bounding boxes at a minimum.

extensions.image_organizer module

class cvjson.extensions.image_organizer.Image_Organizer(cvj)

Bases: cvjson.cvj.CVJ

combine_subfolders()

This method combines the subfolders created by partition_images() in to one big folder of images. Though this method should be able to work on any source folder as long as there are only images in the subfolders.

This method is primarily used when the training algorithm can only hand a source directory of images and cannot walk the subfolders.

convert_images(convert_to='.png', save_internal=False)

This method converts the images found in the image folder path that has been supplied to the CVJ object that this method is being called from. This method uses opencv to handle the conversion.

Parameters:
  • convert_to (string) –
    (Default = .png)
    This is the image type that this method will convert the images to.
  • = bool (save_internal) – (Default = False) This will save the internal json data to the supplied json path to the object calling this method if this is true. Refer to “Introduction to the CVJ” for more information about supplying a json path to the CVJ object
gather_negatives_to_internal_json(negative_image_path=None, move_images=False)

This method is used to gather any negatives generated from the Painter class and store them in the internal json data of this object. If the move_images varibale is True then it will also move the images that were tranferred to the image folder path that is set inside the object. Not the negative_image_path that has been supplied to this method. If the negative_image_path is none then this object will look in the image folder path.

Parameters:
  • negative_image_path (string) – (Default = None) This is the path to a image folder that has negative sample type images created by the Painter class.
  • move_images (bool) –
    (Defaul = False)
    The variable is used to tell the object if the images that are in the negative image folder path should be moved to the image folder path that has been set with cvj_obj.image_folder_path = your/actual/images/to/use
move_images(list_of_images=None, new_image_path=None, from_this=False, transfer=False, set_new_path=False)

This method just moves the images that are associated with the json to the newly supplied path. The path must be a directory, however if the directory does not exist then it will make one for you. The object then sets the image_folder_path attribute to the new location of the images.

Parameters:
  • list_of_images (list) –
    (Default = None)
    This is a list of image filepaths just in case there is a need to move images in a list to a new place.
  • new_image_path (string) – (Default = None) The directory to move the images from or to
  • from_this (bool) –
    (Default = False)
    This tells this method if the images should be coming from the image directory supplied using cvj_obj.image_folder_path = “path/to/images”. Or if it is coming to the image location supplied with cvj_obj.image_folder_path.
  • transfer (bool) –
    (Default = False)
    This tells the method if there is a transfer of images from the list_of_images variable and have nothing to do with the internal json data and tranfers them to the new location supplied with new_image_path.
  • set_new_path (bool) –
    (Default = False)
    This variable is used to set the image folder path of this object instance to the new image folder path that has been supplied.
Returns:

string
Return type:

The supplied directory path.
partition_images(K)

This method partitions the images in to K subfolders in the image directory that has already been supplied to the object. This does not partition them in any specific order is just rotates through the new folders it creates and places an image in each one until it is done. This is mainly used when the folder size is too big to view or has some other weird behaviours when there are too many images consolidated.

See “Introduction to the Image Organizer” for more details on setting the folder path. Just remember the image_folder_path attribute must be set.

Parameters:K (int) – The number of folders the images should be partitioned in to.

extensions.image_tester module

class cvjson.extensions.image_tester.Image_Tester(cvj, image_folder_path=None)

Bases: cvjson.cvj.CVJ

The Image_Tester class takes the json Handle and can perform checks and clean the images

remove_unreadable_files(save=False)

This method just scans the images supplied to the object with the image folder path using opencv. It goes through and opens each file and if there is an error opening the file then it removes that from the json file and moves the unreadable file

NOTE: The path needs to be associated with the json path that was supplied for this to work.

Parameters:None –

extensions.painter module

class cvjson.extensions.painter.Painter(cvj, image_folder_path=None)

Bases: cvjson.cvj.CVJ

The Painter class is used to do painting on the images such as inpainting and hopefully outpainting. This class relies on GMIC

Generation method enums:
  • INPAINT, using the multiscale patch based inpainted method when generating data.
Sample type enums:
  • POSITIVE, deals with positives samples
  • NEGATIVE, deals with negatives samples
  • ALL, deals with both sample types
ALL = 9000
INPAINT = 0
NEGATIVE = 0
POSTIVE = 1
clear_checkpoints()

This method clears out any pickled checkpoints found in the checkpoints folder that is near where the script was calling a generation method.

generate_negatives(save_directory=None, n_cores=1, padding=0, generation_method=0)

This method generates negative examples using the generation_method selected. This means that the images supplied will be used to generate new images without any postive examples on the returned images. This can be done via inpainting only as of right now

TODO: Make more methods to generate negative examples

Parameters:
  • img_path (string) – (Default value = None) A path to the image directory
  • n_cores (int) – (Default value = 1) The number of cores that should be used for generation. Normally higher means faster, but more RAM used.
  • padding (int) – (Default value = 0) How much padding there should be in the image. This should be relatively high to not get out of bounds errors
  • generation_method (int) – (Default value = INPAINT) This is a value that determines the type of algorithm that will be used to generate the negative examples. This is also part of the enums that are associated with the class so since one will most likely be using this outside of the class.

Example

from cvjson.cvjson.cvj import CVJ
from cvjson.extensions.painter import Painter

json_path = "path/to/your/json"

cvj_obj = CVJ(json_path)
painter_obj = Painter(cvj_obj)

# Since we imported the Painter class from the painter module we can use the class variable INPAINT with Painter.INPAINT
painter_obj.generate_negative_backgrounds(self, save_directory='.', n_cores=3, padding=3000, generation_method=Painter.INPAINT):
Returns:list – The only return value is named “negative_filepaths”. This value is a list of all the filepaths to the generated images that are negative sample types.
Return type:list
generate_postives(save_directory=None, n_cores=1, padding=0, generation_method=0)

This method generates negative examples using the generation_method selected. This means that the images supplied will be used to generate new images without any postive examples on the returned images. This can be done via inpainting only as of right now

TODO: Make more methods to generate negative examples

Parameters:
  • img_path (string) – (Default value = None) A path to the image directory
  • n_cores (int) – (Default value = 1) The number of cores that should be used for generation. Normally higher means faster, but more RAM used.
  • padding (int) – (Default value = 0) How much padding there should be in the image. This should be relatively high to not get out of bounds errors
  • generation_method (int) – (Default value = INPAINT) This is a value that determines the type of algorithm that will be used to generate the negative examples. This is also part of the enums/class variable that are associated with the class so since one will most likely be using this outside of the class.

Example

from cvjson.cvjson.cvj import CVJ
from cvjson.extensions.painter import Painter

json_path = "path/to/your/json"

cvj_obj = CVJ(json_path)
painter_obj = Painter(cvj_obj)

# Since we imported the Painter class from the painter module we can use the class variable INPAINT with Painter.INPAINT
painter_obj.generate_negative_backgrounds(self, save_directory='.', n_cores=3, padding=3000, generation_method=Painter.INPAINT):
Returns:list – The only return value is named “positive_filepaths”. This value is a list of all the filepaths to the generated images that are positive sample types.
Return type:list
load_checkpoint()

Note

This is not working as of right now, but is in the generation code. This is one of the methods that will be corrected in the near future.

This method loads the checkpoints which are pickled lists of filepaths for completed images or currently being worked on images.

This is useful for sharing data between processes or if an interruption occurs during the generation, then the program can recover its position of what images are left to work on.

remove_generated(sample_type)
Parameters:sample_type (int) – This is an enum/class variable associated with the Painter class. This variable is used to determine what generated sample types will be removed from the image directory where the generated images are located. Each generated image should have a beginning name start of “negative” or “positive”
Returns:list – This is the list of filepaths of images that are being removed.
Return type:list
start_checkpoint()

Note

This is not working as of right now, but is in the generation code. This is one of the methods that will be corrected in the near future.

This method starts the checkpoints by creating the checkpoints folder and creating empty pickle files.

update_checkpoint()

Note

This is not working as of right now, but is in the generation code. This is one of the methods that will be corrected in the near future.

This method updates the checkpoints in the the checkpoints folder. These checkpoints are the lists of filepaths for images completed or currently being workedon by GMIC.

extensions.visualizer module

class cvjson.extensions.visualizer.Visualizer(cvj)

Bases: cvjson.cvj.CVJ

The Visualizer class takes the json Handle and can perform checks and clean the images

visualize_bboxes()