Information

Do adversarial neural networks exist in the brain?

Do adversarial neural networks exist in the brain?



We are searching data for your request:

Forums and discussions:
Manuals and reference books:
Data from registers:
Wait the end of the search in all databases.
Upon completion, a link will appear to access the found materials.

A recent development in Artificial Neural Networks for machine learning applications is the use of Adversarial Networks (see this paper for an example). Adversarial Networks is a network architecture where one network learns to "fool" the other.

Does this type of adversarial learning exist in the brain at a neuronal level?


AI Interpret Brain Data Produces Personally Attractive Images

Developers have succeeded in creating an AI to read our subjective thoughts of what makes looks attractive. This system illustrated this knowledge by its capability to produce new images that were found attractive to people. The outcomes can be used, for example, in decision-making, modeling preferences, and recognizing unconscious attitudes.

Developers at the University of Copenhagen and the University of Helsinki examined whether a computer would be competent to recognize the facial expression we think attractive and, based on this, form new images meeting our standards. The developers’ used AI to evaluate brain signals and merged the resulting brain-computer interface with a generative model of artificial faces. This allowed the computer to create facial images that appealed to individual decisions.

“Previously, we developed models that could recognize and regulate image characteristics, such as hair tone. Though, people mainly agree on who is blond and who laughs. Attractiveness is a difficult subject of research, as it is associated with social and emotional factors that are likely to play unconscious roles in our individual choices. Indeed, we find it challenging to describe what it is that makes something, or someone, beautiful: Beauty is in the eye of the beholder,” says Docent Michiel Spapé, Psychology and Logopedics department, University of Helsinki.

A computer created facial images that appealed to individual preferences. Credit: COGNITIVE COMPUTING -TUTKIMUSRYHMÄ

Choices Revealed By the Brain

At first, the developers provided GAN (generative adversarial neural network): It is the task of producing hundreds of artificial pictures. Thirty volunteers saw pictures one by one. Then, they were asked to pay attention to the faces in images they found attractive. While electroencephalography (EEG) is used to collect their brain responses.

“It worked similarly to the dating app Tinder: the volunteers ‘swiped right’ when crossed to an attractive face. Here, volunteers did not have to do anything but look at the images and swiped left or right. We marked their instantaneous brain reaction to the images,” Spapé explains.

The developers examined the EEG results with machine learning methods, combining individual EEG results through a brain-computer interface to a generative neural network.

“A brain-computer interface can read users’ views on the attractiveness of a series of images. By understanding their views, the AI system interprets brain responses. The generative neural network illustrating the face pictures can together compose an entirely new face picture by combining with what a particular person finds attractive,” says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To examine the efficacy of their modeling, the developers produced new pictures for each volunteer, predicting they would find them individually attractive. Experimenting with them in a double-blind manner against matched controls, they found that the new images resembled the choices of the subjects with an accuracy of over 80%.

“The research shows that we can generate images that match personal choice by combining an artificial neural network to brain responses. Succeeding in evaluating attractiveness is very important, as this is such a touching, emotional part of the stimuli. The computer concept has been quite successful at classifying pictures based on objective models. By bringing in brain responses to the mix, we show it is conceivable to recognize and create pictures based on subjective characteristics, like personal taste,” Spapé explains.

Eventually, the research may serve society by improving the potential for computers to learn and frequently understand individual preferences, by the interaction between AI solutions and brain-computer interfaces.


4 Results

4.1 Experiment setup

In this section, we evaluate the proposed benign perturbation calibrater based on CIFAR10 data set, using MobileNet howard2017mobilenets , VGG16 simonyan2014very , and ResNet18 he2016deep

models (also referred as main models). Our implementation is based on PyTorch and we use 4 different data augmentation functions in PyTorch to define the so called A/B scenario setting. We first assume that training data and test data for CIFAR10 are identically and independently distributed. We refer data that are transformed via random crop resize and random horizontal flip as scenario A, and refer data that are transformed via random rotation and random color jittering as scenario B. For further comparison, we also derive B1 and B2 as scenario B under increasing strength of data transformations. For scenario B1, maximum rotation is 15, maximum change of brightness is 0.8, maximum change of contrast is 0.8, maximum change of saturation is 0.8. And for scenario B2, the maximum rotation is 20, maximum change of brightness is 2, maximum change of contrast is 2 and maximum change of saturation is 2. For scenario A, we only use PyTorch’s default settings. All main models are trained under scenario A in training data, and tested under scenario B1/B2 in test data. The calibraters are trained under scenario B1/B2 in training data and used to support main models under scenario B1/B2 in test data. The backbone of the generative model is consisted of 3 down-sampling convolutional layers, followed by a number of residual blocks and 3 up-sampling convolutional layers. The sizes of the calibraters are chosen to be one tenth of the main models. To control the size of the calibrater, we vary the number of channels within residual blocks and number of residual blocks. For example, to have a calibrater that has one tenth parameter the size of MobileNet, we use 2 residual blocks with 18 channels. The solver chosen for training main models and calibraters is ADAM

. Training for calibraters takes 200 epochs and the initial learning rate is set to be 0.0002 before 50 epochs, 0.0001 at 50-100 epochs, 0.00005 at 100-150 epochs, and 0.00002 at 150-200 epochs. During training of calibraters, main models are only used for feed-forward.

4.2 Robustness improvement brought by reverse adversarial examples method

Results on uncompressed models: Table 2 shows the robustness increase brought by the proposed method. In Table 2 Column 3 lists models’ test accuracy on clean test data without any data augmentation. When models are tested under scenario B1 and B2 in test data, models’ accuracy drop about 20% to 40%, as shown in Column 4 in Table 2 ). We consider random rotation and random color jittering as common environmental factors in real world and this result demonstrates that deep learning models lack of robustness in our settings. Column 5 shows that by using the proposed calibrater, we can achieve 4.8% to 19.66% accuracy boost (Column 5 in Table 2 ), with only little overhead brought by the calibrater (Column 6&7 in Table 2 ).

4.3 Robustness improvement on compressed models

Results on compressed models: In this subsection, we demonstrate the effectiveness of the proposed calibrater on compressed models. Main models’ weights are quantized to 2/3 bits using deep compression han2015deep . To ensure that overhead brought by our calibraters is still low compared to the main models, we quantize them to 8 bits without any quantization training. Note that the calibraters’ training can naturally leverage the accelerated inference speed of main models. In comparison between Column 4 in Table 2 and Column 3 in Table 3 , we observe that quantized models suffer significantly more accuracy drop under scenario B2 in test data. We consider this phenomenon as an indicator that model compression hurts models’ robustness in our settings. Table 3 Column 5 shows that when deployed the calibraters, the models’ performance on B2 improves significantly.


Beauty Is in the Brain: AI Generates Attractive Images From Brain Data

Researchers have succeeded in making an AI understand our subjective notions of what makes faces attractive. The device demonstrated this knowledge by its ability to create new portraits on its own that were tailored to be found personally attractive to individuals. The results can be utilised, for example, in modelling preferences and decision-making as well as potentially identifying unconscious attitudes.

Researchers at the University of Helsinki and University of Copenhagen investigated whether a computer would be able to identify the facial features we consider attractive and, based on this, create new images matching our criteria. The researchers used artificial intelligence to interpret brain signals and combined the resulting brain-computer interface with a generative model of artificial faces. This enabled the computer to create facial images that appealed to individual preferences.

"In our previous studies, we designed models that could identify and control simple portrait features, such as hair colour and emotion. However, people largely agree on who is blond and who smiles. Attractiveness is a more challenging subject of study, as it is associated with cultural and psychological factors that likely play unconscious roles in our individual preferences. Indeed, we often find it very hard to explain what it is exactly that makes something, or someone, beautiful: Beauty is in the eye of the beholder," says Senior Researcher and Docent Michiel Spapé from the Department of Psychology and Logopedics, University of Helsinki.

The study, which combines computer science and psychology, was published in February in the IEEE Transactions in Affective Computing journal.

Preferences exposed by the brain

Initially, the researchers gave a generative adversarial neural network (GAN) the task of creating hundreds of artificial portraits. The images were shown, one at a time, to 30 volunteers who were asked to pay attention to faces they found attractive while their brain responses were recorded via electroencephalography (EEG).

"It worked a bit like the dating app Tinder: the participants 'swiped right' when coming across an attractive face. Here, however, they did not have to do anything but look at the images. We measured their immediate brain response to the images," Spapé explains.

The researchers analysed the EEG data with machine learning techniques, connecting individual EEG data through a brain-computer interface to a generative neural network.

"A brain-computer interface such as this is able to interpret users' opinions on the attractiveness of a range of images. By interpreting their views, the AI model interpreting brain responses and the generative neural network modelling the face images can together produce an entirely new face image by combining what a particular person finds attractive," says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To test the validity of their modelling, the researchers generated new portraits for each participant, predicting they would find them personally attractive. Testing them in a double-blind procedure against matched controls, they found that the new images matched the preferences of the subjects with an accuracy of over 80%.

"The study demonstrates that we are capable of generating images that match personal preference by connecting an artificial neural network to brain responses. Succeeding in assessing attractiveness is especially significant, as this is such a poignant, psychological property of the stimuli. Computer vision has thus far been very successful at categorising images based on objective patterns. By bringing in brain responses to the mix, we show it is possible to detect and generate images based on psychological properties, like personal taste," Spapé explains.

Potential for exposing unconscious attitudes

Ultimately, the study may benefit society by advancing the capacity for computers to learn and increasingly understand subjective preferences, through interaction between AI solutions and brain-computer interfaces.

"If this is possible in something that is as personal and subjective as attractiveness, we may also be able to look into other cognitive functions such as perception and decision-making. Potentially, we might gear the device towards identifying stereotypes or implicit bias and better understand individual differences," says Spapé.

Reference: Spape M, Davis K, Kangassalo L, et al. Brain-computer interface for generating personally attractive images. IEEE Trans. Affect. Comput. doi: 10.1109/TAFFC.2021.3059043.

This article has been republished from the following materials. Note: material may have been edited for length and content. For further information, please contact the cited source.


Summary: Merry GAN-mas: Introduction to NVIDIA StyleGAN2 ADA

Generative Adversarial Neural Networks (GANs) are a type of neural network that can generate random “fake” images based on a training set of real images. GANs were introduced by Ian Goodfellow in his 2014 paper. GANs trained to produce human faces have received much media attention since the release of NVIDIA StyleGAN in 2018. Websites like Which Face is Real and This Person Does Not Exist demonstrate the amazing capabilities of NVIDIA StyleGAN. In this article I will explore the latest GAN technology, NVIDIA StyleGAN2 and demonstrate how to train it to produce holiday images.

The first step is to obtain a set of images to train the GAN. I created a Python utility called pyimgdata that you can use to download images from Flickr and perform other preprocessing. Flickr is a great place to obtain images and is used by many GAN paper authors, such as NVIDIA. Flickr is beneficial because it has an API to obtain images and contains license information for each upload. When building a dataset of images, it is generally advisable to use only images published by their authors with a permissive license.

My Flickr download utility makes use of a configuration file, such as the following:

This script downloads the results of the specified search into the specified path. The filenames will have the specified prefix. I specify all licenses because I do not intend to publish this image list. This actually brings up an open issue in copyright law. If a neural network learns from copyrighted and produces new work, is the AI bound to the original copyright? Similarly, is a human musician who listens to copyrighted music beholden to the copyright owner for inspiration that the music had on the musician’s brain? For the purposes of copyright, I consider my GAN and its images to be a derivative work.


A neural network learns when it should not be trusted

Increasingly, artificial intelligence systems known as deep learning neural networks are used to inform decisions vital to human health and safety, such as in autonomous driving or medical diagnosis. These networks are good at recognizing patterns in large, complex datasets to aid in decision-making. But how do we know they're correct? Alexander Amini and his colleagues at MIT and Harvard University wanted to find out.

They've developed a quick way for a neural network to crunch data, and output not just a prediction but also the model's confidence level based on the quality of the available data. The advance might save lives, as deep learning is already being deployed in the real world today. A network's level of certainty can be the difference between an autonomous vehicle determining that "it's all clear to proceed through the intersection" and "it's probably clear, so stop just in case."

Current methods of uncertainty estimation for neural networks tend to be computationally expensive and relatively slow for split-second decisions. But Amini's approach, dubbed "deep evidential regression," accelerates the process and could lead to safer outcomes. "We need the ability to not only have high-performance models, but also to understand when we cannot trust those models," says Amini, a PhD student in Professor Daniela Rus' group at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).

"This idea is important and applicable broadly. It can be used to assess products that rely on learned models. By estimating the uncertainty of a learned model, we also learn how much error to expect from the model, and what missing data could improve the model," says Rus.

Amini will present the research at next month's NeurIPS conference, along with Rus, who is the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science, director of CSAIL, and deputy dean of research for the MIT Stephen A. Schwarzman College of Computing and graduate students Wilko Schwarting of MIT and Ava Soleimany of MIT and Harvard.

Efficient uncertainty

After an up-and-down history, deep learning has demonstrated remarkable performance on a variety of tasks, in some cases even surpassing human accuracy. And nowadays, deep learning seems to go wherever computers go. It fuels search engine results, social media feeds, and facial recognition. "We've had huge successes using deep learning," says Amini. "Neural networks are really good at knowing the right answer 99 percent of the time." But 99 percent won't cut it when lives are on the line.

"One thing that has eluded researchers is the ability of these models to know and tell us when they might be wrong," says Amini. "We really care about that 1 percent of the time, and how we can detect those situations reliably and efficiently."

Neural networks can be massive, sometimes brimming with billions of parameters. So it can be a heavy computational lift just to get an answer, let alone a confidence level. Uncertainty analysis in neural networks isn't new. But previous approaches, stemming from Bayesian deep learning, have relied on running, or sampling, a neural network many times over to understand its confidence. That process takes time and memory, a luxury that might not exist in high-speed traffic.

The researchers devised a way to estimate uncertainty from only a single run of the neural network. They designed the network with bulked up output, producing not only a decision but also a new probabilistic distribution capturing the evidence in support of that decision. These distributions, termed evidential distributions, directly capture the model's confidence in its prediction. This includes any uncertainty present in the underlying input data, as well as in the model's final decision. This distinction can signal whether uncertainty can be reduced by tweaking the neural network itself, or whether the input data are just noisy.

Confidence check

To put their approach to the test, the researchers started with a challenging computer vision task. They trained their neural network to analyze a monocular color image and estimate a depth value (i.e. distance from the camera lens) for each pixel. An autonomous vehicle might use similar calculations to estimate its proximity to a pedestrian or to another vehicle, which is no simple task.

Their network's performance was on par with previous state-of-the-art models, but it also gained the ability to estimate its own uncertainty. As the researchers had hoped, the network projected high uncertainty for pixels where it predicted the wrong depth. "It was very calibrated to the errors that the network makes, which we believe was one of the most important things in judging the quality of a new uncertainty estimator," Amini says.

To stress-test their calibration, the team also showed that the network projected higher uncertainty for "out-of-distribution" data -- completely new types of images never encountered during training. After they trained the network on indoor home scenes, they fed it a batch of outdoor driving scenes. The network consistently warned that its responses to the novel outdoor scenes were uncertain. The test highlighted the network's ability to flag when users should not place full trust in its decisions. In these cases, "if this is a health care application, maybe we don't trust the diagnosis that the model is giving, and instead seek a second opinion," says Amini.

The network even knew when photos had been doctored, potentially hedging against data-manipulation attacks. In another trial, the researchers boosted adversarial noise levels in a batch of images they fed to the network. The effect was subtle -- barely perceptible to the human eye -- but the network sniffed out those images, tagging its output with high levels of uncertainty. This ability to sound the alarm on falsified data could help detect and deter adversarial attacks, a growing concern in the age of deepfakes.

Deep evidential regression is "a simple and elegant approach that advances the field of uncertainty estimation, which is important for robotics and other real-world control systems," says Raia Hadsell, an artificial intelligence researcher at DeepMind who was not involved with the work. "This is done in a novel way that avoids some of the messy aspects of other approaches -- e.g. sampling or ensembles -- which makes it not only elegant but also computationally more efficient -- a winning combination."

Deep evidential regression could enhance safety in AI-assisted decision making. "We're starting to see a lot more of these [neural network] models trickle out of the research lab and into the real world, into situations that are touching humans with potentially life-threatening consequences," says Amini. "Any user of the method, whether it's a doctor or a person in the passenger seat of a vehicle, needs to be aware of any risk or uncertainty associated with that decision." He envisions the system not only quickly flagging uncertainty, but also using it to make more conservative decision making in risky scenarios like an autonomous vehicle approaching an intersection.

"Any field that is going to have deployable machine learning ultimately needs to have reliable uncertainty awareness," he says.

This work was supported, in part, by the National Science Foundation and Toyota Research Institute through the Toyota-CSAIL Joint Research Center.


Beauty is in the brain: AI reads brain data, generates personally attractive images

A computer created facial images that appealed to individual preferences. Credit: Cognitive computing research group

Researchers have succeeded in making an AI understand our subjective notions of what makes faces attractive. The device demonstrated this knowledge by its ability to create new portraits that were tailored to be found personally attractive to individuals. The results can be used, for example, in modeling preferences and decision-making as well as potentially identifying unconscious attitudes.

Researchers at the University of Helsinki and University of Copenhagen investigated whether a computer would be able to identify the facial features we consider attractive and, based on this, create new images matching our criteria. The researchers used artificial intelligence to interpret brain signals and combined the resulting brain-computer interface with a generative model of artificial faces. This enabled the computer to create facial images that appealed to individual preferences.

"In our previous studies, we designed models that could identify and control simple portrait features, such as hair color and emotion. However, people largely agree on who is blond and who smiles. Attractiveness is a more challenging subject of study, as it is associated with cultural and psychological factors that likely play unconscious roles in our individual preferences. Indeed, we often find it very hard to explain what it is exactly that makes something, or someone, beautiful: Beauty is in the eye of the beholder," says Senior Researcher and Docent Michiel Spapé from the Department of Psychology and Logopedics, University of Helsinki.

The study, which combines computer science and psychology, was published in February in the IEEE Transactions in Affective Computing journal.

Preferences exposed by the brain

Initially, the researchers gave a generative adversarial neural network (GAN) the task of creating hundreds of artificial portraits. The images were shown, one at a time, to 30 volunteers who were asked to pay attention to faces they found attractive while their brain responses were recorded via electroencephalography (EEG).

"It worked a bit like the dating app Tinder: The participants 'swiped right' when coming across an attractive face. Here, however, they did not have to do anything but look at the images. We measured their immediate brain response to the images," Spapé explains.

The researchers analyzed the EEG data with machine learning techniques, connecting individual EEG data through a brain-computer interface to a generative neural network.

"A brain-computer interface such as this is able to interpret users' opinions on the attractiveness of a range of images. By interpreting their views, the AI model interpreting brain responses and the generative neural network modeling the face images can together produce an entirely new face image by combining what a particular person finds attractive," says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To test the validity of their modeling, the researchers generated new portraits for each participant, predicting they would find them personally attractive. Testing them in a double-blind procedure against matched controls, they found that the new images matched the preferences of the subjects with an accuracy of over 80%.

"The study demonstrates that we are capable of generating images that match personal preference by connecting an artificial neural network to brain responses. Succeeding in assessing attractiveness is especially significant, as this is such a poignant, psychological property of the stimuli. Computer vision has thus far been very successful at categorizing images based on objective patterns. By bringing in brain responses to the mix, we show it is possible to detect and generate images based on psychological properties, like personal taste," Spapé explains.

Potential for exposing unconscious attitudes

Ultimately, the study may benefit society by advancing the capacity for computers to learn and increasingly understand subjective preferences, through interaction between AI solutions and brain-computer interfaces.

"If this is possible in something that is as personal and subjective as attractiveness, we may also be able to look into other cognitive functions such as perception and decision-making. Potentially, we might gear the device towards identifying stereotypes or implicit bias and better understand individual differences," says Spapé.


Contents

The generative network generates candidates while the discriminative network evaluates them. [1] The contest operates in terms of data distributions. Typically, the generative network learns to map from a latent space to a data distribution of interest, while the discriminative network distinguishes candidates produced by the generator from the true data distribution. The generative network's training objective is to increase the error rate of the discriminative network (i.e., "fool" the discriminator network by producing novel candidates that the discriminator thinks are not synthesized (are part of the true data distribution)). [1] [6]

A known dataset serves as the initial training data for the discriminator. Training it involves presenting it with samples from the training dataset, until it achieves acceptable accuracy. The generator trains based on whether it succeeds in fooling the discriminator. Typically the generator is seeded with randomized input that is sampled from a predefined latent space (e.g. a multivariate normal distribution). Thereafter, candidates synthesized by the generator are evaluated by the discriminator. Independent backpropagation procedures are applied to both networks so that the generator produces better samples, while the discriminator becomes more skilled at flagging synthetic samples. [7] When used for image generation, the generator is typically a deconvolutional neural network, and the discriminator is a convolutional neural network.

GANs often suffer from a "mode collapse" where they fail to generalize properly, missing entire modes from the input data. For example, a GAN trained on the MNIST dataset containing many samples of each digit, might nevertheless timidly omit a subset of the digits from its output. Some researchers perceive the root problem to be a weak discriminative network that fails to notice the pattern of omission, while others assign blame to a bad choice of objective function. Many solutions have been proposed. [8]

GANs are implicit generative models, [9] which means that they do not explicitly model the likelihood function nor provide means for finding the latent variable corresponding to a given sample, unlike alternatives such as Flow-based generative model.

GAN applications have increased rapidly. [10]

Fashion, art and advertising Edit

GANs can be used to generate art The Verge wrote in March 2019 that "The images created by GANs have become the defining look of contemporary AI art." [11] GANs can also be used to inpaint photographs [12] or create photos of imaginary fashion models, with no need to hire a model, photographer or makeup artist, or pay for a studio and transportation. [13]

Science Edit

GANs can improve astronomical images [14] and simulate gravitational lensing for dark matter research. [15] [16] [17] They were used in 2019 to successfully model the distribution of dark matter in a particular direction in space and to predict the gravitational lensing that will occur. [18] [19]

GANs have been proposed as a fast and accurate way of modeling high energy jet formation [20] and modeling showers through calorimeters of high-energy physics experiments. [21] [22] [23] [24] GANs have also been trained to accurately approximate bottlenecks in computationally expensive simulations of particle physics experiments. Applications in the context of present and proposed CERN experiments have demonstrated the potential of these methods for accelerating simulation and/or improving simulation fidelity. [25] [26]

Video games Edit

In 2018, GANs reached the video game modding community, as a method of up-scaling low-resolution 2D textures in old video games by recreating them in 4k or higher resolutions via image training, and then down-sampling them to fit the game's native resolution (with results resembling the supersampling method of anti-aliasing). [27] With proper training, GANs provide a clearer and sharper 2D texture image magnitudes higher in quality than the original, while fully retaining the original's level of details, colors, etc. Known examples of extensive GAN usage include Final Fantasy VIII, Final Fantasy IX, Resident Evil REmake HD Remaster, and Max Payne. [ citation needed ]

Concerns about malicious applications Edit

Concerns have been raised about the potential use of GAN-based human image synthesis for sinister purposes, e.g., to produce fake, possibly incriminating, photographs and videos. [28] GANs can be used to generate unique, realistic profile photos of people who do not exist, in order to automate creation of fake social media profiles. [29]

In 2019 the state of California considered [30] and passed on October 3, 2019 the bill AB-602, which bans the use of human image synthesis technologies to make fake pornography without the consent of the people depicted, and bill AB-730, which prohibits distribution of manipulated videos of a political candidate within 60 days of an election. Both bills were authored by Assembly member Marc Berman and signed by Governor Gavin Newsom. The laws will come into effect in 2020. [31]

DARPA's Media Forensics program studies ways to counteract fake media, including fake media produced using GANs. [32]

Transfer learning Edit

State-of-art transfer learning research use GANs to enforce the alignment of the latent feature space, such as in deep reinforcement learning. [33] This works by feeding the embeddings of the source and target task to the discriminator which tries to guess the context. The resulting loss is then (inversely) backpropogated through the encoder.

Miscellaneous applications Edit

GAN can be used to detect glaucomatous images helping the early diagnosis which is essential to avoid partial or total loss of vision. [34]

GANs that produce photorealistic images can be used to visualize interior design, industrial design, shoes, [35] bags, and clothing items or items for computer games' scenes. [ citation needed ] Such networks were reported to be used by Facebook. [36]

GANs can reconstruct 3D models of objects from images, [37] and model patterns of motion in video. [38]

GANs can be used to age face photographs to show how an individual's appearance might change with age. [39]

GANs can also be used to transfer map styles in cartography [40] or augment street view imagery. [41]

Relevance feedback on GANs can be used to generate images and replace image search systems. [42]

A variation of the GANs is used in training a network to generate optimal control inputs to nonlinear dynamical systems. Where the discriminatory network is known as a critic that checks the optimality of the solution and the generative network is known as an Adaptive network that generates the optimal control. The critic and adaptive network train each other to approximate a nonlinear optimal control. [43]

GANs have been used to visualize the effect that climate change will have on specific houses. [44]

A GAN model called Speech2Face can reconstruct an image of a person's face after listening to their voice. [45]

In 2016 GANs were used to generate new molecules for a variety of protein targets implicated in cancer, inflammation, and fibrosis. In 2019 GAN-generated molecules were validated experimentally all the way into mice. [46] [47]

Whereas the majority of GAN applications are in image processing, the work has also been done with time-series data. For example, recurrent GANs (R-GANs) have been used to generate energy data for machine learning. [48]

The most direct inspiration for GANs was noise-contrastive estimation, [49] which uses the same loss function as GANs and which Goodfellow studied during his PhD in 2010–2014.

Other people had similar ideas but did not develop them similarly. An idea involving adversarial networks was published in a 2010 blog post by Olli Niemitalo. [50] This idea was never implemented and did not involve stochasticity in the generator and thus was not a generative model. It is now known as a conditional GAN or cGAN. [51] An idea similar to GANs was used to model animal behavior by Li, Gauci and Gross in 2013. [52]

Adversarial machine learning has other uses besides generative modeling and can be applied to models other than neural networks. In control theory, adversarial learning based on neural networks was used in 2006 to train robust controllers in a game theoretic sense, by alternating the iterations between a minimizer policy, the controller, and a maximizer policy, the disturbance. [53] [54]

In 2017, a GAN was used for image enhancement focusing on realistic textures rather than pixel-accuracy, producing a higher image quality at high magnification. [55] In 2017, the first faces were generated. [56] These were exhibited in February 2018 at the Grand Palais. [57] [58] Faces generated by StyleGAN [59] in 2019 drew comparisons with deepfakes. [60] [61] [62]

Beginning in 2017, GAN technology began to make its presence felt in the fine arts arena with the appearance of a newly developed implementation which was said to have crossed the threshold of being able to generate unique and appealing abstract paintings, and thus dubbed a "CAN", for "creative adversarial network". [63] A GAN system was used to create the 2018 painting Edmond de Belamy, which sold for US$432,500. [64] An early 2019 article by members of the original CAN team discussed further progress with that system, and gave consideration as well to the overall prospects for an AI-enabled art. [65]

In May 2019, researchers at Samsung demonstrated a GAN-based system that produces videos of a person speaking, given only a single photo of that person. [66]

In August 2019, a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment was created for neural melody generation from lyrics using conditional GAN-LSTM (refer to sources at GitHub AI Melody Generation from Lyrics). [67]

In May 2020, Nvidia researchers taught an AI system (termed "GameGAN") to recreate the game of Pac-Man simply by watching it being played. [68] [69]

Bidirectional GAN Edit

While the standard GAN model learns a mapping from a latent space to the data distribution, inverse models such as Bidirectional GAN (BiGAN) [70] and Adversarial Autoencoders [71] also learn a mapping from data to the latent space. This inverse mapping allows real or generated data examples to be projected back into the latent space, similar to the encoder of a variational autoencoder. Applications of bidirectional models include semi-supervised learning, [72] interpretable machine learning, [73] and neural machine translation. [74]


In this post, you discovered a gentle introduction to Generative Adversarial Networks, or GANs.

  • Context for GANs, including supervised vs. unsupervised learning and discriminative vs. generative modeling.
  • GANs are an architecture for automatically training a generative model by treating the unsupervised problem as supervised and using both a generative and a discriminative model.
  • GANs provide a path to sophisticated domain-specific data augmentation and a solution to problems that require a generative solution, such as image-to-image translation.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.


Beauty is in the brain: AI reads brain data, generates personally attractive images

Researchers have succeeded in making an AI understand our subjective notions of what makes faces attractive. The device demonstrated this knowledge by its ability to create new portraits on its own that were tailored to be found personally attractive to individuals. The results can be utilised, for example, in modelling preferences and decision-making as well as potentially identifying unconscious attitudes.

Researchers at the University of Helsinki and University of Copenhagen investigated whether a computer would be able to identify the facial features we consider attractive and, based on this, create new images matching our criteria. The researchers used artificial intelligence to interpret brain signals and combined the resulting brain-computer interface with a generative model of artificial faces. This enabled the computer to create facial images that appealed to individual preferences.

"In our previous studies, we designed models that could identify and control simple portrait features, such as hair colour and emotion. However, people largely agree on who is blond and who smiles. Attractiveness is a more challenging subject of study, as it is associated with cultural and psychological factors that likely play unconscious roles in our individual preferences. Indeed, we often find it very hard to explain what it is exactly that makes something, or someone, beautiful: Beauty is in the eye of the beholder," says Senior Researcher and Docent Michiel Spapé from the Department of Psychology and Logopedics, University of Helsinki.

The study, which combines computer science and psychology, was published in February in the IEEE Transactions in Affective Computing journal.

Preferences exposed by the brain

Initially, the researchers gave a generative adversarial neural network (GAN) the task of creating hundreds of artificial portraits. The images were shown, one at a time, to 30 volunteers who were asked to pay attention to faces they found attractive while their brain responses were recorded via electroencephalography (EEG).

"It worked a bit like the dating app Tinder: the participants 'swiped right' when coming across an attractive face. Here, however, they did not have to do anything but look at the images. We measured their immediate brain response to the images," Spapé explains.

The researchers analysed the EEG data with machine learning techniques, connecting individual EEG data through a brain-computer interface to a generative neural network.

"A brain-computer interface such as this is able to interpret users' opinions on the attractiveness of a range of images. By interpreting their views, the AI model interpreting brain responses and the generative neural network modelling the face images can together produce an entirely new face image by combining what a particular person finds attractive," says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To test the validity of their modelling, the researchers generated new portraits for each participant, predicting they would find them personally attractive. Testing them in a double-blind procedure against matched controls, they found that the new images matched the preferences of the subjects with an accuracy of over 80%.

"The study demonstrates that we are capable of generating images that match personal preference by connecting an artificial neural network to brain responses. Succeeding in assessing attractiveness is especially significant, as this is such a poignant, psychological property of the stimuli. Computer vision has thus far been very successful at categorising images based on objective patterns. By bringing in brain responses to the mix, we show it is possible to detect and generate images based on psychological properties, like personal taste," Spapé explains.

Potential for exposing unconscious attitudes

Ultimately, the study may benefit society by advancing the capacity for computers to learn and increasingly understand subjective preferences, through interaction between AI solutions and brain-computer interfaces.

"If this is possible in something that is as personal and subjective as attractiveness, we may also be able to look into other cognitive functions such as perception and decision-making. Potentially, we might gear the device towards identifying stereotypes or implicit bias and better understand individual differences," says Spapé.


A neural network learns when it should not be trusted

Increasingly, artificial intelligence systems known as deep learning neural networks are used to inform decisions vital to human health and safety, such as in autonomous driving or medical diagnosis. These networks are good at recognizing patterns in large, complex datasets to aid in decision-making. But how do we know they're correct? Alexander Amini and his colleagues at MIT and Harvard University wanted to find out.

They've developed a quick way for a neural network to crunch data, and output not just a prediction but also the model's confidence level based on the quality of the available data. The advance might save lives, as deep learning is already being deployed in the real world today. A network's level of certainty can be the difference between an autonomous vehicle determining that "it's all clear to proceed through the intersection" and "it's probably clear, so stop just in case."

Current methods of uncertainty estimation for neural networks tend to be computationally expensive and relatively slow for split-second decisions. But Amini's approach, dubbed "deep evidential regression," accelerates the process and could lead to safer outcomes. "We need the ability to not only have high-performance models, but also to understand when we cannot trust those models," says Amini, a PhD student in Professor Daniela Rus' group at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).

"This idea is important and applicable broadly. It can be used to assess products that rely on learned models. By estimating the uncertainty of a learned model, we also learn how much error to expect from the model, and what missing data could improve the model," says Rus.

Amini will present the research at next month's NeurIPS conference, along with Rus, who is the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science, director of CSAIL, and deputy dean of research for the MIT Stephen A. Schwarzman College of Computing and graduate students Wilko Schwarting of MIT and Ava Soleimany of MIT and Harvard.

Efficient uncertainty

After an up-and-down history, deep learning has demonstrated remarkable performance on a variety of tasks, in some cases even surpassing human accuracy. And nowadays, deep learning seems to go wherever computers go. It fuels search engine results, social media feeds, and facial recognition. "We've had huge successes using deep learning," says Amini. "Neural networks are really good at knowing the right answer 99 percent of the time." But 99 percent won't cut it when lives are on the line.

"One thing that has eluded researchers is the ability of these models to know and tell us when they might be wrong," says Amini. "We really care about that 1 percent of the time, and how we can detect those situations reliably and efficiently."

Neural networks can be massive, sometimes brimming with billions of parameters. So it can be a heavy computational lift just to get an answer, let alone a confidence level. Uncertainty analysis in neural networks isn't new. But previous approaches, stemming from Bayesian deep learning, have relied on running, or sampling, a neural network many times over to understand its confidence. That process takes time and memory, a luxury that might not exist in high-speed traffic.

The researchers devised a way to estimate uncertainty from only a single run of the neural network. They designed the network with bulked up output, producing not only a decision but also a new probabilistic distribution capturing the evidence in support of that decision. These distributions, termed evidential distributions, directly capture the model's confidence in its prediction. This includes any uncertainty present in the underlying input data, as well as in the model's final decision. This distinction can signal whether uncertainty can be reduced by tweaking the neural network itself, or whether the input data are just noisy.

Confidence check

To put their approach to the test, the researchers started with a challenging computer vision task. They trained their neural network to analyze a monocular color image and estimate a depth value (i.e. distance from the camera lens) for each pixel. An autonomous vehicle might use similar calculations to estimate its proximity to a pedestrian or to another vehicle, which is no simple task.

Their network's performance was on par with previous state-of-the-art models, but it also gained the ability to estimate its own uncertainty. As the researchers had hoped, the network projected high uncertainty for pixels where it predicted the wrong depth. "It was very calibrated to the errors that the network makes, which we believe was one of the most important things in judging the quality of a new uncertainty estimator," Amini says.

To stress-test their calibration, the team also showed that the network projected higher uncertainty for "out-of-distribution" data -- completely new types of images never encountered during training. After they trained the network on indoor home scenes, they fed it a batch of outdoor driving scenes. The network consistently warned that its responses to the novel outdoor scenes were uncertain. The test highlighted the network's ability to flag when users should not place full trust in its decisions. In these cases, "if this is a health care application, maybe we don't trust the diagnosis that the model is giving, and instead seek a second opinion," says Amini.

The network even knew when photos had been doctored, potentially hedging against data-manipulation attacks. In another trial, the researchers boosted adversarial noise levels in a batch of images they fed to the network. The effect was subtle -- barely perceptible to the human eye -- but the network sniffed out those images, tagging its output with high levels of uncertainty. This ability to sound the alarm on falsified data could help detect and deter adversarial attacks, a growing concern in the age of deepfakes.

Deep evidential regression is "a simple and elegant approach that advances the field of uncertainty estimation, which is important for robotics and other real-world control systems," says Raia Hadsell, an artificial intelligence researcher at DeepMind who was not involved with the work. "This is done in a novel way that avoids some of the messy aspects of other approaches -- e.g. sampling or ensembles -- which makes it not only elegant but also computationally more efficient -- a winning combination."

Deep evidential regression could enhance safety in AI-assisted decision making. "We're starting to see a lot more of these [neural network] models trickle out of the research lab and into the real world, into situations that are touching humans with potentially life-threatening consequences," says Amini. "Any user of the method, whether it's a doctor or a person in the passenger seat of a vehicle, needs to be aware of any risk or uncertainty associated with that decision." He envisions the system not only quickly flagging uncertainty, but also using it to make more conservative decision making in risky scenarios like an autonomous vehicle approaching an intersection.

"Any field that is going to have deployable machine learning ultimately needs to have reliable uncertainty awareness," he says.

This work was supported, in part, by the National Science Foundation and Toyota Research Institute through the Toyota-CSAIL Joint Research Center.


In this post, you discovered a gentle introduction to Generative Adversarial Networks, or GANs.

  • Context for GANs, including supervised vs. unsupervised learning and discriminative vs. generative modeling.
  • GANs are an architecture for automatically training a generative model by treating the unsupervised problem as supervised and using both a generative and a discriminative model.
  • GANs provide a path to sophisticated domain-specific data augmentation and a solution to problems that require a generative solution, such as image-to-image translation.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.


AI Interpret Brain Data Produces Personally Attractive Images

Developers have succeeded in creating an AI to read our subjective thoughts of what makes looks attractive. This system illustrated this knowledge by its capability to produce new images that were found attractive to people. The outcomes can be used, for example, in decision-making, modeling preferences, and recognizing unconscious attitudes.

Developers at the University of Copenhagen and the University of Helsinki examined whether a computer would be competent to recognize the facial expression we think attractive and, based on this, form new images meeting our standards. The developers’ used AI to evaluate brain signals and merged the resulting brain-computer interface with a generative model of artificial faces. This allowed the computer to create facial images that appealed to individual decisions.

“Previously, we developed models that could recognize and regulate image characteristics, such as hair tone. Though, people mainly agree on who is blond and who laughs. Attractiveness is a difficult subject of research, as it is associated with social and emotional factors that are likely to play unconscious roles in our individual choices. Indeed, we find it challenging to describe what it is that makes something, or someone, beautiful: Beauty is in the eye of the beholder,” says Docent Michiel Spapé, Psychology and Logopedics department, University of Helsinki.

A computer created facial images that appealed to individual preferences. Credit: COGNITIVE COMPUTING -TUTKIMUSRYHMÄ

Choices Revealed By the Brain

At first, the developers provided GAN (generative adversarial neural network): It is the task of producing hundreds of artificial pictures. Thirty volunteers saw pictures one by one. Then, they were asked to pay attention to the faces in images they found attractive. While electroencephalography (EEG) is used to collect their brain responses.

“It worked similarly to the dating app Tinder: the volunteers ‘swiped right’ when crossed to an attractive face. Here, volunteers did not have to do anything but look at the images and swiped left or right. We marked their instantaneous brain reaction to the images,” Spapé explains.

The developers examined the EEG results with machine learning methods, combining individual EEG results through a brain-computer interface to a generative neural network.

“A brain-computer interface can read users’ views on the attractiveness of a series of images. By understanding their views, the AI system interprets brain responses. The generative neural network illustrating the face pictures can together compose an entirely new face picture by combining with what a particular person finds attractive,” says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To examine the efficacy of their modeling, the developers produced new pictures for each volunteer, predicting they would find them individually attractive. Experimenting with them in a double-blind manner against matched controls, they found that the new images resembled the choices of the subjects with an accuracy of over 80%.

“The research shows that we can generate images that match personal choice by combining an artificial neural network to brain responses. Succeeding in evaluating attractiveness is very important, as this is such a touching, emotional part of the stimuli. The computer concept has been quite successful at classifying pictures based on objective models. By bringing in brain responses to the mix, we show it is conceivable to recognize and create pictures based on subjective characteristics, like personal taste,” Spapé explains.

Eventually, the research may serve society by improving the potential for computers to learn and frequently understand individual preferences, by the interaction between AI solutions and brain-computer interfaces.


4 Results

4.1 Experiment setup

In this section, we evaluate the proposed benign perturbation calibrater based on CIFAR10 data set, using MobileNet howard2017mobilenets , VGG16 simonyan2014very , and ResNet18 he2016deep

models (also referred as main models). Our implementation is based on PyTorch and we use 4 different data augmentation functions in PyTorch to define the so called A/B scenario setting. We first assume that training data and test data for CIFAR10 are identically and independently distributed. We refer data that are transformed via random crop resize and random horizontal flip as scenario A, and refer data that are transformed via random rotation and random color jittering as scenario B. For further comparison, we also derive B1 and B2 as scenario B under increasing strength of data transformations. For scenario B1, maximum rotation is 15, maximum change of brightness is 0.8, maximum change of contrast is 0.8, maximum change of saturation is 0.8. And for scenario B2, the maximum rotation is 20, maximum change of brightness is 2, maximum change of contrast is 2 and maximum change of saturation is 2. For scenario A, we only use PyTorch’s default settings. All main models are trained under scenario A in training data, and tested under scenario B1/B2 in test data. The calibraters are trained under scenario B1/B2 in training data and used to support main models under scenario B1/B2 in test data. The backbone of the generative model is consisted of 3 down-sampling convolutional layers, followed by a number of residual blocks and 3 up-sampling convolutional layers. The sizes of the calibraters are chosen to be one tenth of the main models. To control the size of the calibrater, we vary the number of channels within residual blocks and number of residual blocks. For example, to have a calibrater that has one tenth parameter the size of MobileNet, we use 2 residual blocks with 18 channels. The solver chosen for training main models and calibraters is ADAM

. Training for calibraters takes 200 epochs and the initial learning rate is set to be 0.0002 before 50 epochs, 0.0001 at 50-100 epochs, 0.00005 at 100-150 epochs, and 0.00002 at 150-200 epochs. During training of calibraters, main models are only used for feed-forward.

4.2 Robustness improvement brought by reverse adversarial examples method

Results on uncompressed models: Table 2 shows the robustness increase brought by the proposed method. In Table 2 Column 3 lists models’ test accuracy on clean test data without any data augmentation. When models are tested under scenario B1 and B2 in test data, models’ accuracy drop about 20% to 40%, as shown in Column 4 in Table 2 ). We consider random rotation and random color jittering as common environmental factors in real world and this result demonstrates that deep learning models lack of robustness in our settings. Column 5 shows that by using the proposed calibrater, we can achieve 4.8% to 19.66% accuracy boost (Column 5 in Table 2 ), with only little overhead brought by the calibrater (Column 6&7 in Table 2 ).

4.3 Robustness improvement on compressed models

Results on compressed models: In this subsection, we demonstrate the effectiveness of the proposed calibrater on compressed models. Main models’ weights are quantized to 2/3 bits using deep compression han2015deep . To ensure that overhead brought by our calibraters is still low compared to the main models, we quantize them to 8 bits without any quantization training. Note that the calibraters’ training can naturally leverage the accelerated inference speed of main models. In comparison between Column 4 in Table 2 and Column 3 in Table 3 , we observe that quantized models suffer significantly more accuracy drop under scenario B2 in test data. We consider this phenomenon as an indicator that model compression hurts models’ robustness in our settings. Table 3 Column 5 shows that when deployed the calibraters, the models’ performance on B2 improves significantly.


Contents

The generative network generates candidates while the discriminative network evaluates them. [1] The contest operates in terms of data distributions. Typically, the generative network learns to map from a latent space to a data distribution of interest, while the discriminative network distinguishes candidates produced by the generator from the true data distribution. The generative network's training objective is to increase the error rate of the discriminative network (i.e., "fool" the discriminator network by producing novel candidates that the discriminator thinks are not synthesized (are part of the true data distribution)). [1] [6]

A known dataset serves as the initial training data for the discriminator. Training it involves presenting it with samples from the training dataset, until it achieves acceptable accuracy. The generator trains based on whether it succeeds in fooling the discriminator. Typically the generator is seeded with randomized input that is sampled from a predefined latent space (e.g. a multivariate normal distribution). Thereafter, candidates synthesized by the generator are evaluated by the discriminator. Independent backpropagation procedures are applied to both networks so that the generator produces better samples, while the discriminator becomes more skilled at flagging synthetic samples. [7] When used for image generation, the generator is typically a deconvolutional neural network, and the discriminator is a convolutional neural network.

GANs often suffer from a "mode collapse" where they fail to generalize properly, missing entire modes from the input data. For example, a GAN trained on the MNIST dataset containing many samples of each digit, might nevertheless timidly omit a subset of the digits from its output. Some researchers perceive the root problem to be a weak discriminative network that fails to notice the pattern of omission, while others assign blame to a bad choice of objective function. Many solutions have been proposed. [8]

GANs are implicit generative models, [9] which means that they do not explicitly model the likelihood function nor provide means for finding the latent variable corresponding to a given sample, unlike alternatives such as Flow-based generative model.

GAN applications have increased rapidly. [10]

Fashion, art and advertising Edit

GANs can be used to generate art The Verge wrote in March 2019 that "The images created by GANs have become the defining look of contemporary AI art." [11] GANs can also be used to inpaint photographs [12] or create photos of imaginary fashion models, with no need to hire a model, photographer or makeup artist, or pay for a studio and transportation. [13]

Science Edit

GANs can improve astronomical images [14] and simulate gravitational lensing for dark matter research. [15] [16] [17] They were used in 2019 to successfully model the distribution of dark matter in a particular direction in space and to predict the gravitational lensing that will occur. [18] [19]

GANs have been proposed as a fast and accurate way of modeling high energy jet formation [20] and modeling showers through calorimeters of high-energy physics experiments. [21] [22] [23] [24] GANs have also been trained to accurately approximate bottlenecks in computationally expensive simulations of particle physics experiments. Applications in the context of present and proposed CERN experiments have demonstrated the potential of these methods for accelerating simulation and/or improving simulation fidelity. [25] [26]

Video games Edit

In 2018, GANs reached the video game modding community, as a method of up-scaling low-resolution 2D textures in old video games by recreating them in 4k or higher resolutions via image training, and then down-sampling them to fit the game's native resolution (with results resembling the supersampling method of anti-aliasing). [27] With proper training, GANs provide a clearer and sharper 2D texture image magnitudes higher in quality than the original, while fully retaining the original's level of details, colors, etc. Known examples of extensive GAN usage include Final Fantasy VIII, Final Fantasy IX, Resident Evil REmake HD Remaster, and Max Payne. [ citation needed ]

Concerns about malicious applications Edit

Concerns have been raised about the potential use of GAN-based human image synthesis for sinister purposes, e.g., to produce fake, possibly incriminating, photographs and videos. [28] GANs can be used to generate unique, realistic profile photos of people who do not exist, in order to automate creation of fake social media profiles. [29]

In 2019 the state of California considered [30] and passed on October 3, 2019 the bill AB-602, which bans the use of human image synthesis technologies to make fake pornography without the consent of the people depicted, and bill AB-730, which prohibits distribution of manipulated videos of a political candidate within 60 days of an election. Both bills were authored by Assembly member Marc Berman and signed by Governor Gavin Newsom. The laws will come into effect in 2020. [31]

DARPA's Media Forensics program studies ways to counteract fake media, including fake media produced using GANs. [32]

Transfer learning Edit

State-of-art transfer learning research use GANs to enforce the alignment of the latent feature space, such as in deep reinforcement learning. [33] This works by feeding the embeddings of the source and target task to the discriminator which tries to guess the context. The resulting loss is then (inversely) backpropogated through the encoder.

Miscellaneous applications Edit

GAN can be used to detect glaucomatous images helping the early diagnosis which is essential to avoid partial or total loss of vision. [34]

GANs that produce photorealistic images can be used to visualize interior design, industrial design, shoes, [35] bags, and clothing items or items for computer games' scenes. [ citation needed ] Such networks were reported to be used by Facebook. [36]

GANs can reconstruct 3D models of objects from images, [37] and model patterns of motion in video. [38]

GANs can be used to age face photographs to show how an individual's appearance might change with age. [39]

GANs can also be used to transfer map styles in cartography [40] or augment street view imagery. [41]

Relevance feedback on GANs can be used to generate images and replace image search systems. [42]

A variation of the GANs is used in training a network to generate optimal control inputs to nonlinear dynamical systems. Where the discriminatory network is known as a critic that checks the optimality of the solution and the generative network is known as an Adaptive network that generates the optimal control. The critic and adaptive network train each other to approximate a nonlinear optimal control. [43]

GANs have been used to visualize the effect that climate change will have on specific houses. [44]

A GAN model called Speech2Face can reconstruct an image of a person's face after listening to their voice. [45]

In 2016 GANs were used to generate new molecules for a variety of protein targets implicated in cancer, inflammation, and fibrosis. In 2019 GAN-generated molecules were validated experimentally all the way into mice. [46] [47]

Whereas the majority of GAN applications are in image processing, the work has also been done with time-series data. For example, recurrent GANs (R-GANs) have been used to generate energy data for machine learning. [48]

The most direct inspiration for GANs was noise-contrastive estimation, [49] which uses the same loss function as GANs and which Goodfellow studied during his PhD in 2010–2014.

Other people had similar ideas but did not develop them similarly. An idea involving adversarial networks was published in a 2010 blog post by Olli Niemitalo. [50] This idea was never implemented and did not involve stochasticity in the generator and thus was not a generative model. It is now known as a conditional GAN or cGAN. [51] An idea similar to GANs was used to model animal behavior by Li, Gauci and Gross in 2013. [52]

Adversarial machine learning has other uses besides generative modeling and can be applied to models other than neural networks. In control theory, adversarial learning based on neural networks was used in 2006 to train robust controllers in a game theoretic sense, by alternating the iterations between a minimizer policy, the controller, and a maximizer policy, the disturbance. [53] [54]

In 2017, a GAN was used for image enhancement focusing on realistic textures rather than pixel-accuracy, producing a higher image quality at high magnification. [55] In 2017, the first faces were generated. [56] These were exhibited in February 2018 at the Grand Palais. [57] [58] Faces generated by StyleGAN [59] in 2019 drew comparisons with deepfakes. [60] [61] [62]

Beginning in 2017, GAN technology began to make its presence felt in the fine arts arena with the appearance of a newly developed implementation which was said to have crossed the threshold of being able to generate unique and appealing abstract paintings, and thus dubbed a "CAN", for "creative adversarial network". [63] A GAN system was used to create the 2018 painting Edmond de Belamy, which sold for US$432,500. [64] An early 2019 article by members of the original CAN team discussed further progress with that system, and gave consideration as well to the overall prospects for an AI-enabled art. [65]

In May 2019, researchers at Samsung demonstrated a GAN-based system that produces videos of a person speaking, given only a single photo of that person. [66]

In August 2019, a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment was created for neural melody generation from lyrics using conditional GAN-LSTM (refer to sources at GitHub AI Melody Generation from Lyrics). [67]

In May 2020, Nvidia researchers taught an AI system (termed "GameGAN") to recreate the game of Pac-Man simply by watching it being played. [68] [69]

Bidirectional GAN Edit

While the standard GAN model learns a mapping from a latent space to the data distribution, inverse models such as Bidirectional GAN (BiGAN) [70] and Adversarial Autoencoders [71] also learn a mapping from data to the latent space. This inverse mapping allows real or generated data examples to be projected back into the latent space, similar to the encoder of a variational autoencoder. Applications of bidirectional models include semi-supervised learning, [72] interpretable machine learning, [73] and neural machine translation. [74]


Beauty is in the brain: AI reads brain data, generates personally attractive images

A computer created facial images that appealed to individual preferences. Credit: Cognitive computing research group

Researchers have succeeded in making an AI understand our subjective notions of what makes faces attractive. The device demonstrated this knowledge by its ability to create new portraits that were tailored to be found personally attractive to individuals. The results can be used, for example, in modeling preferences and decision-making as well as potentially identifying unconscious attitudes.

Researchers at the University of Helsinki and University of Copenhagen investigated whether a computer would be able to identify the facial features we consider attractive and, based on this, create new images matching our criteria. The researchers used artificial intelligence to interpret brain signals and combined the resulting brain-computer interface with a generative model of artificial faces. This enabled the computer to create facial images that appealed to individual preferences.

"In our previous studies, we designed models that could identify and control simple portrait features, such as hair color and emotion. However, people largely agree on who is blond and who smiles. Attractiveness is a more challenging subject of study, as it is associated with cultural and psychological factors that likely play unconscious roles in our individual preferences. Indeed, we often find it very hard to explain what it is exactly that makes something, or someone, beautiful: Beauty is in the eye of the beholder," says Senior Researcher and Docent Michiel Spapé from the Department of Psychology and Logopedics, University of Helsinki.

The study, which combines computer science and psychology, was published in February in the IEEE Transactions in Affective Computing journal.

Preferences exposed by the brain

Initially, the researchers gave a generative adversarial neural network (GAN) the task of creating hundreds of artificial portraits. The images were shown, one at a time, to 30 volunteers who were asked to pay attention to faces they found attractive while their brain responses were recorded via electroencephalography (EEG).

"It worked a bit like the dating app Tinder: The participants 'swiped right' when coming across an attractive face. Here, however, they did not have to do anything but look at the images. We measured their immediate brain response to the images," Spapé explains.

The researchers analyzed the EEG data with machine learning techniques, connecting individual EEG data through a brain-computer interface to a generative neural network.

"A brain-computer interface such as this is able to interpret users' opinions on the attractiveness of a range of images. By interpreting their views, the AI model interpreting brain responses and the generative neural network modeling the face images can together produce an entirely new face image by combining what a particular person finds attractive," says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To test the validity of their modeling, the researchers generated new portraits for each participant, predicting they would find them personally attractive. Testing them in a double-blind procedure against matched controls, they found that the new images matched the preferences of the subjects with an accuracy of over 80%.

"The study demonstrates that we are capable of generating images that match personal preference by connecting an artificial neural network to brain responses. Succeeding in assessing attractiveness is especially significant, as this is such a poignant, psychological property of the stimuli. Computer vision has thus far been very successful at categorizing images based on objective patterns. By bringing in brain responses to the mix, we show it is possible to detect and generate images based on psychological properties, like personal taste," Spapé explains.

Potential for exposing unconscious attitudes

Ultimately, the study may benefit society by advancing the capacity for computers to learn and increasingly understand subjective preferences, through interaction between AI solutions and brain-computer interfaces.

"If this is possible in something that is as personal and subjective as attractiveness, we may also be able to look into other cognitive functions such as perception and decision-making. Potentially, we might gear the device towards identifying stereotypes or implicit bias and better understand individual differences," says Spapé.


Beauty is in the brain: AI reads brain data, generates personally attractive images

Researchers have succeeded in making an AI understand our subjective notions of what makes faces attractive. The device demonstrated this knowledge by its ability to create new portraits on its own that were tailored to be found personally attractive to individuals. The results can be utilised, for example, in modelling preferences and decision-making as well as potentially identifying unconscious attitudes.

Researchers at the University of Helsinki and University of Copenhagen investigated whether a computer would be able to identify the facial features we consider attractive and, based on this, create new images matching our criteria. The researchers used artificial intelligence to interpret brain signals and combined the resulting brain-computer interface with a generative model of artificial faces. This enabled the computer to create facial images that appealed to individual preferences.

"In our previous studies, we designed models that could identify and control simple portrait features, such as hair colour and emotion. However, people largely agree on who is blond and who smiles. Attractiveness is a more challenging subject of study, as it is associated with cultural and psychological factors that likely play unconscious roles in our individual preferences. Indeed, we often find it very hard to explain what it is exactly that makes something, or someone, beautiful: Beauty is in the eye of the beholder," says Senior Researcher and Docent Michiel Spapé from the Department of Psychology and Logopedics, University of Helsinki.

The study, which combines computer science and psychology, was published in February in the IEEE Transactions in Affective Computing journal.

Preferences exposed by the brain

Initially, the researchers gave a generative adversarial neural network (GAN) the task of creating hundreds of artificial portraits. The images were shown, one at a time, to 30 volunteers who were asked to pay attention to faces they found attractive while their brain responses were recorded via electroencephalography (EEG).

"It worked a bit like the dating app Tinder: the participants 'swiped right' when coming across an attractive face. Here, however, they did not have to do anything but look at the images. We measured their immediate brain response to the images," Spapé explains.

The researchers analysed the EEG data with machine learning techniques, connecting individual EEG data through a brain-computer interface to a generative neural network.

"A brain-computer interface such as this is able to interpret users' opinions on the attractiveness of a range of images. By interpreting their views, the AI model interpreting brain responses and the generative neural network modelling the face images can together produce an entirely new face image by combining what a particular person finds attractive," says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To test the validity of their modelling, the researchers generated new portraits for each participant, predicting they would find them personally attractive. Testing them in a double-blind procedure against matched controls, they found that the new images matched the preferences of the subjects with an accuracy of over 80%.

"The study demonstrates that we are capable of generating images that match personal preference by connecting an artificial neural network to brain responses. Succeeding in assessing attractiveness is especially significant, as this is such a poignant, psychological property of the stimuli. Computer vision has thus far been very successful at categorising images based on objective patterns. By bringing in brain responses to the mix, we show it is possible to detect and generate images based on psychological properties, like personal taste," Spapé explains.

Potential for exposing unconscious attitudes

Ultimately, the study may benefit society by advancing the capacity for computers to learn and increasingly understand subjective preferences, through interaction between AI solutions and brain-computer interfaces.

"If this is possible in something that is as personal and subjective as attractiveness, we may also be able to look into other cognitive functions such as perception and decision-making. Potentially, we might gear the device towards identifying stereotypes or implicit bias and better understand individual differences," says Spapé.


Beauty Is in the Brain: AI Generates Attractive Images From Brain Data

Researchers have succeeded in making an AI understand our subjective notions of what makes faces attractive. The device demonstrated this knowledge by its ability to create new portraits on its own that were tailored to be found personally attractive to individuals. The results can be utilised, for example, in modelling preferences and decision-making as well as potentially identifying unconscious attitudes.

Researchers at the University of Helsinki and University of Copenhagen investigated whether a computer would be able to identify the facial features we consider attractive and, based on this, create new images matching our criteria. The researchers used artificial intelligence to interpret brain signals and combined the resulting brain-computer interface with a generative model of artificial faces. This enabled the computer to create facial images that appealed to individual preferences.

"In our previous studies, we designed models that could identify and control simple portrait features, such as hair colour and emotion. However, people largely agree on who is blond and who smiles. Attractiveness is a more challenging subject of study, as it is associated with cultural and psychological factors that likely play unconscious roles in our individual preferences. Indeed, we often find it very hard to explain what it is exactly that makes something, or someone, beautiful: Beauty is in the eye of the beholder," says Senior Researcher and Docent Michiel Spapé from the Department of Psychology and Logopedics, University of Helsinki.

The study, which combines computer science and psychology, was published in February in the IEEE Transactions in Affective Computing journal.

Preferences exposed by the brain

Initially, the researchers gave a generative adversarial neural network (GAN) the task of creating hundreds of artificial portraits. The images were shown, one at a time, to 30 volunteers who were asked to pay attention to faces they found attractive while their brain responses were recorded via electroencephalography (EEG).

"It worked a bit like the dating app Tinder: the participants 'swiped right' when coming across an attractive face. Here, however, they did not have to do anything but look at the images. We measured their immediate brain response to the images," Spapé explains.

The researchers analysed the EEG data with machine learning techniques, connecting individual EEG data through a brain-computer interface to a generative neural network.

"A brain-computer interface such as this is able to interpret users' opinions on the attractiveness of a range of images. By interpreting their views, the AI model interpreting brain responses and the generative neural network modelling the face images can together produce an entirely new face image by combining what a particular person finds attractive," says Academy Research Fellow and Associate Professor Tuukka Ruotsalo, who heads the project.

To test the validity of their modelling, the researchers generated new portraits for each participant, predicting they would find them personally attractive. Testing them in a double-blind procedure against matched controls, they found that the new images matched the preferences of the subjects with an accuracy of over 80%.

"The study demonstrates that we are capable of generating images that match personal preference by connecting an artificial neural network to brain responses. Succeeding in assessing attractiveness is especially significant, as this is such a poignant, psychological property of the stimuli. Computer vision has thus far been very successful at categorising images based on objective patterns. By bringing in brain responses to the mix, we show it is possible to detect and generate images based on psychological properties, like personal taste," Spapé explains.

Potential for exposing unconscious attitudes

Ultimately, the study may benefit society by advancing the capacity for computers to learn and increasingly understand subjective preferences, through interaction between AI solutions and brain-computer interfaces.

"If this is possible in something that is as personal and subjective as attractiveness, we may also be able to look into other cognitive functions such as perception and decision-making. Potentially, we might gear the device towards identifying stereotypes or implicit bias and better understand individual differences," says Spapé.

Reference: Spape M, Davis K, Kangassalo L, et al. Brain-computer interface for generating personally attractive images. IEEE Trans. Affect. Comput. doi: 10.1109/TAFFC.2021.3059043.

This article has been republished from the following materials. Note: material may have been edited for length and content. For further information, please contact the cited source.


Summary: Merry GAN-mas: Introduction to NVIDIA StyleGAN2 ADA

Generative Adversarial Neural Networks (GANs) are a type of neural network that can generate random “fake” images based on a training set of real images. GANs were introduced by Ian Goodfellow in his 2014 paper. GANs trained to produce human faces have received much media attention since the release of NVIDIA StyleGAN in 2018. Websites like Which Face is Real and This Person Does Not Exist demonstrate the amazing capabilities of NVIDIA StyleGAN. In this article I will explore the latest GAN technology, NVIDIA StyleGAN2 and demonstrate how to train it to produce holiday images.

The first step is to obtain a set of images to train the GAN. I created a Python utility called pyimgdata that you can use to download images from Flickr and perform other preprocessing. Flickr is a great place to obtain images and is used by many GAN paper authors, such as NVIDIA. Flickr is beneficial because it has an API to obtain images and contains license information for each upload. When building a dataset of images, it is generally advisable to use only images published by their authors with a permissive license.

My Flickr download utility makes use of a configuration file, such as the following:

This script downloads the results of the specified search into the specified path. The filenames will have the specified prefix. I specify all licenses because I do not intend to publish this image list. This actually brings up an open issue in copyright law. If a neural network learns from copyrighted and produces new work, is the AI bound to the original copyright? Similarly, is a human musician who listens to copyrighted music beholden to the copyright owner for inspiration that the music had on the musician’s brain? For the purposes of copyright, I consider my GAN and its images to be a derivative work.