I'm dealing with a problem where I couldn't find enough dataset(images) to feed into my deep neural network for training.
I was so inspired by the paper Generative Adversarial Text to Image Synthesis published by Scott Reed et al. on Generative Adversarial Networks.
I was curious to know that, can I use available small dataset as an input to a GAN model and generate a much bigger dataset to deal with deeper network models?
Will it be good enough?
This is unlikely to add much beyond your direct data collection efforts.
The quality of current GAN outputs (as of 2017) will not be high enough. The images produced by a GAN are typically small and can have unusual/ambiguous details and odd distortions. In the paper you linked, the images generated by the system from a sentence have believable blocks of colour given the subject matter, but without the sentence priming you what to expect most of them are not recognisable as any specific subject.
GANs with a less ambitious purpose than generating images from sentences (which is despite my criticism above, a truly remarkable feat IMO) should produce closer to photo-realistic images. But their scope will be less and probably not include your desired image type. Also, typically the output size is small e.g. 64x64 or 128x128*, and there are still enough distortions and ambiguities that original ground truth photos would be far preferable.
The GAN is itself limited by training library available - it will not do well if you attempt to generate images outside of the scope of its training data. The results shown in the research paper of course focus on the domain supplied by the training data. But you cannot just feed any sentence into this model and expect a result that would be useful elsewhere.
If you find a GAN that has been trained on a suitable data set for your problem, then you are most likely better off trying to source the same data directly for your project.
If you are facing a problem with limited ground truth data, then maybe a better approach to using a GAN would be to use a pre-trained classifier such as VGG-19 or Inception v5, replace the last few fully-connected layers, and fine tune it on your data. Here is an example of doing that using Keras library in Python - other examples can be found with searches like "fine tune CNN image classifier".
* State-of-the art GANs have got better since I posted this answer. A research team at Nvidia has had remarkable success creating 1024x1024 photo-realistic images. However, this does not change the other points in my answer. GANs are not a reliable source of images for image classification tasks, except maybe for sub-tasks of whatever the GAN has already been trained on and is able to generate conditionally (or maybe more trivially, to provide source data for "other" categories in classifiers).
I have the exact same issue with a DNN that I am currently building. Taking my data set and synthesizing new data with a GAN seems like a great idea. But the GAN itself will only learn to output images with the same image variance and standard deviations as was learned in the training set. So your newly generated data will simply represent more permutations of the same sample distribution. This will help your NN train better on the same distribution, therefore it may lead to greater over training.
Just from a purely theoretical perspective this cannot be possible.
Any given training dataset represents a certain amount of information about the structure of a certain space. If you train a GAN on this dataset, it will only ever learn from the information represented by that dataset. The data synthesized by the GAN can never be from a larger space than the original data, for the simple reason: Where would this information be supposed to come from? If it wasn't in the original dataset, then it also cannot be in the synthesized data from the GAN.
If you train a neural network to convergence on a dataset, that neural network will learn whatever structure the dataset contains. Any artificial training data synthesized by a GAN will add no new information. That idea should be straight forward.
Actually it is possible, to augment a small Dataset with GANs to improve it and it will also increase the Performance of Classification Networks as you can read here.
GANs are capable of learning e.g. intermediate shapes which are not involved in the original dateset but still are valid. So synthetic images indeed can improve dataset size and improve CNN classification accuracy.
