Google AI Blog: August 2018

Understanding Performance Fluctuations in Quantum Processors

Friday, August 31, 2018

Posted by Paul V. Klimov, Research Scientist, Google AI Quantum TeamGoogle AI Quantum teamsuperconducting electrical circuitsquantum bitsmodest processor sizesFluctuations of Energy-Relaxation Times in Superconducting QubitsPhysical Review Letters energy relaxation timesT1T1

Left: A quantum processor similar to the one that was used to investigate qubit performance fluctuations. One qubit is highlighted in blue. Right: One qubit’s energy-relaxation time “T1” plotted as a function of it’s operating frequency and time. We see energy-relaxation hotspots, which our data suggest are due to material defects (black arrowheads). The motion of these hotspots into and out-of resonance with the qubit are responsible for the most significant energy-relaxation fluctuations. Note that these data were taken over a frequency band with an above-average density of defects.

still puzzles researchersthermodynamics arguments

Teaching the Google Assistant to be Multilingual

Thursday, August 30, 2018

Posted by Johan Schalkwyk, VP and Ignacio Lopez Moreno, Engineer, Google Speech123launching

The Google Assistant is now able to identify the language, interpret the query and provide a response using the right language without the user having to touch the Assistant settings.

Identifying Multiple LanguagesUnderstanding Multiple LanguagesOptimizing Multilingual RecognitionIdentifying Multiple Languages¹45recurrent neural networksUnderstanding Multiple Languagesset an alarm for 6pmset an alarm6pmany given pairspeech recognition systems

Schematic of our multilingual speech recognition system used by the Google Assistant versus the standard monolingual speech recognition system. A ranking algorithm is used to select the best recognition hypotheses from two monolingual speech recognizer using relevant information about the user and the incremental langID results.

Optimizing Multilingual Recognitionrandom forestBilingual to Trilingualall
1 It is typically acknowledged that spoken language recognition is remarkably more challenging than text-based language identification where, relatively simple techniques based on dictionaries can do a good job. The time/frequency patterns of spoken words are difficult to compare, spoken words can be more difficult to delimit as they can be spoken without pause and at different paces and microphones may record background noise in addition to speech.^↩

Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research

Monday, August 27, 2018

Posted by Pablo Samuel Castro, Research Software Developer and Marc G. Bellemare, Research Scientist, Google Brain TeamReinforcement learningDQNAlphaGoAlphaGo ZeroOpen AI Fivelarge-scale distributed trainindistributional methodsrobotic manipulationteaching robots to visually self-adaptTensorflow-based frameworkreward-motivated behaviour in the brainset of colabsEase of Usewell-documentedArcade Learning EnvironmentDQNC51Rainbow agentImplicit Quantile Network agentInternational Conference on Machine LearningReproducibilityMachado et al. (2018)Benchmarkingfull training dataJSON data filesa website

The training runs for our 4 agents on Seaquest. The x-axis represents iterations, where each iteration is 1 million game frames (4.5 hours of real-time play); the y-axis is the average score obtained per play. The shaded areas show confidence intervals from 5 independent runs.

Tensorboarddownloads sectiongithub repoAcknowledgementsThis project was only possible thanks to several collaborations at Google. The core team includes Marc G. Bellemare, Pablo Samuel Castro, Carles Gelada, Subhodeep Moitra and Saurabh Kumar. We also extend a special thanks to Sergio Guadamarra, Ofir Nachum, Yifan Wu, Clare Lyle, Liam Fedus, Kelvin Xu, Emilio Parisoto, Hado van Hasselt, Georg Ostrovski and Will Dabney, and the many people at Google who helped us test it out.

Moving Beyond Translation with the Universal Transformer

Wednesday, August 15, 2018

Posted by Stephan Gouws, Research Scientist, Google Brain Team and Mostafa Dehghani, University of Amsterdam PhD student and Google Research Internthe Transformerrecurrent neural networkssequentiallyrecurrenceabcabcabcNeural GPUNeural Turing MachineUniversal TransformersTuring completeparallel-in-time recurrence

The Universal Transformer repeatedly refines a series of vector representations (shown as h₁ to h_m) for each position of the sequence in parallel, by combining information from different positions using self-attention and applying a recurrent transition function. Arrows denote dependencies between operations.

adaptive computation mechanismI arrived at the bank after crossing the riverbankIriverbanksinglesequencedifferent transformationsbAbI linguistic reasoning taskLAMBADA language modeling taskBLEU¹hereAcknowledgementsThis research was conducted by Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Additional thanks go to Ashish Vaswani, Douglas Eck, and David Dohan for their fruitful comments and inspiration.
1 A translation quality benchmark widely used in the machine translation community, computed on the standard WMT newstest2014 English to German translation test data set.^↩

The Machine Learning Behind Android Smart Linkify

Thursday, August 9, 2018

Posted by Lukas Zilka, Software Engineer, Google AI, Zürichwe launched Android 9 PieSmart Linkify

Smart Text SelectionTextClassifier APITensorFlowTensorFlow LiteFlatBuffershereFinding EntitiesConfirmation number: 857-555-3556feedforward neural networks

For the given text string, the first network assigns low scores to non-entities and a high score for the candidate that correctly selects the whole phone number.

And call 857 555 3556tomorrow.857 555 3556And call 857 555 3556 tomorrowAnd

And call 857 555 3556 tomorrow.857 555 3556And call 857 555 3556 tomorrowAndTextual Features

Given a candidate entity span, we extract: Left context: five words before the entity, Entity start: first three words of the entity, Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many), Right context: five words after the entity, Entity content: bag of words inside the entity and Entity length: size of the entity in number of words. They are then concatenated together and fed as an input to the neural network.

Character N-grams. Instead of using the standard word embedding technique for representing words, which keeps a separate vector for each word in the model and thus would be infeasible for mobile devices because of their large storage size, we use the hashed charactergram embedding. This technique represents the word as a set of all character subsequences of certain length. We use lengths 1 to 5. These strings are additionally hashed and mapped to a fixed number of buckets (see here for more details on the technique). As a result, the final model only stores vectors for each of the hash buckets, not each word/character subsequence, and can be kept small. The embedding matrix for the hashed charactergrams that we use has 20,000 buckets and 12 dimensions.

A binary feature that indicates whether the word starts with a capital letter. This is important for the network to know because the capitalization in postal addresses is quite distinct, and helps the networks to discriminate.

A Training DatasetSchema.orgConfirmation number:ID:Making it Work

Quantizing the embedding matrix to 8 bits. We found that we could reduce the size of the model almost 4x without compromising the performance, by quantizing the embedding matrix values to 8-bit integers.

Sharing embedding matrices between the selection and classification networks. This brings almost no loss and makes the model 2x smaller.

Varying the size of the context before/after the entities. On mobile screens text is often short, with not enough context, so the network needs to be exposed to this during training as well.

Creating artificial negative examples out of the positive ones for the classification network. For example for the positive example: “call me 857 555-3556 today” with a label “phone” we generate “call me 857 555-3556 today” as a negative example with a label “other”. This teaches the classification network to be more precise about the entity span. Without doing this, the network would be merely a detector whether there is a phone number somewhere in the input, regardless of the span.

Internationalization is ImportantNext Stepsnext Thursdayin 3 weeksopen-source as part of Android framework

MnasNet: Towards Automating the Design of Mobile Machine Learning Models

Tuesday, August 7, 2018

Posted by Mingxing Tan, Software Engineer, Google Brain TeamConvolutional neural networksMobileNetMobileNetV2AutoML neural architecture searchMnasNet: Platform-Aware Neural Architecture Search for Mobilereinforcement learningMobileNetV2NASNetFLOPSRNNTensorFlow Litemulti-objective optimizationPareto optimal

Overall flow of our automated neural architecture search approach for Mobile.

Our MnasNet network, sampled from the novel factorized hierarchical search space,illustrating the layer diversity throughout the network architecture.

ImageNetCOCO

ImageNet Accuracy and Inference Latency comparison. MnasNets are our models.

MobileNetV2NASNetsqueeze-and-excitationResNet-50SSD300AcknowledgementsSpecial thanks to the co-authors of the paper Bo Chen, Quoc V. Le, Ruoming Pang and Vijay Vasudevan. We’d also like to thank Andrew Howard, Barret Zoph, Dmitry Kalenichenko, Guiheng Zhou, Jeff Dean, Mark Sandler, Megan Kacholia, Sheng Li, Vishy Tirumalashetty, Wen Wang, Xiaoqiang Zheng and Yifeng Lu for their help, and the TensorFlow Lite and Google Brain teams.

Sep	OCT	Nov
	14
2018	2019	2020

Blog

Understanding Performance Fluctuations in Quantum Processors

Teaching the Google Assistant to be Multilingual

Introducing a New Framework for Flexible and Reproducible Reinforcement Learning Research

Moving Beyond Translation with the Universal Transformer

The Machine Learning Behind Android Smart Linkify

MnasNet: Towards Automating the Design of Mobile Machine Learning Models

Labels

Archive

Feed