PhD Positions

Reinforcement learning, which is considered by some as the vanguard of ML, is supposed to imitate the behavior of living beings by an adaptation to the environment via trial-and-error. Living beings, however, do not blindly try out actions — they reason, although, depending on species, maybe in a somewhat primitive way. In most reinforcement learning applications, we rather deal with more or less safe environments where we can first train the neural nets, e.g., actor, critic. model estimator, with sufficient data, and then run them on-field. The cost of a failure during the learning phase is relatively low in such scenarios. Consider AlphaZero, the famous chess bot, based on reinforcement learning. Evidently, a loss in a blitz game is just yet another entry in the data set. Open AI Gym, a popular Python tool of reinforcement learning, learns the agents by trial-and-error also. But what should one do when dealing with real-world applications where no luxury of multiple trials and errors is available? Aircraft, autonomous cars, industrial equipment, robots – there are tons of cases where “playground” conditions are not provided. On the contrary, one faces strict requirements on safety and operational regulations. Traditionally, all of these have been addressed by control and system theory with its long-established machinery of model-based policies with built-in safety constraints.

The challenge of this project is to develop a reinforcement learning approach free of the necessity of a long training phase and capable of more or less “hot start” like traditional control. Here is a question to think about: can we develop a hybrid of reinforcement learning and model-based, predictive engines to possess the intelligence of machine learning and strict guarantees like traditional control?

For the time being, reinforcement learning is barely applicable in industry precisely due to the limitations described. If we want to roll out a really commercializable algorithm – we need to address the above challenge.

We may take the general reinforcement learning setup and investigate how it can be augmented so as to become fully online-capable. There are several ways of tackling this. A particular one is to build reinforcement learning on top of some nominal controller that guarantees safety by default and launch the learning in a virtual environment where such a controller is ambient. Another way is to balance data-driven elements of reinforcement learning with predictive controls.

• Machine Learning
• Control Engineering
• Computer Science and Data Science
• Applied Mathematics

Other eligibility criteria:

• Basic knowledge in Machine Learning, Dynamical Systems, Mathematical Modeling, System Identification, Stochastic Processes
• Programming skills: Python, MATLAB
• High interest in Reinforcement Learning
• Previous experience in numerical simulation of dynamic systems
• Motivation to complete a PhD within 4 years

Most of the research and development in natural language processing is based on machine learning methods, which require a large amount of annotated resources (training corpora). Both industry and scientific groups regularly encounter a shortage of such resources when conducting their research and development. Annotation of text corpora is a very time-consuming process, so only a few languages have a substantial amount of resources suitable for machine learning. For most other languages, including Russian, the lack of resources is a very acute problem. The situation is even more severe if it is necessary to analyze texts from specific subject areas, for example, from clinical medicine or biomedicine. In such cases, corpus annotation is very expensive, because instead of simple crowdsourcing, it is necessary to attract highly qualified specialists (physicians and scientists). In this case, annotation with the help of active training can significantly reduce costs and accelerate the development of text analysis tools.

Active learning iteratively selects unlabeled objects and shows them to a human annotator. It shows only objects that are considered “informative”. This helps to reduce annotation redundancy and avoid unimportant objects, for example, outliers. One of the essential components of active learning methods is query strategies of unlabeled training examples that are shown to human annotators. Applying active learning commonly reduces the amount of annotation by several times but retains the high quality of the trained model. In the near future, this technology is expected to become widely used in both industrial and research projects related to machine learning, and, in particular, in projects related to natural language processing.

The state-of-the-art methods for natural language processing are based on deep neural models with a large number of parameters that take advantage of transfer learning: ELMo, BERT, RoBERTa, ELECTRA, inter alia. The project aims is to develop methods of active learning that are in alignment with the abilities of these recently proposed models. The research can be conducted along with one of the two main directions:

• Trainable policies as AL strategies based on reinforcement learning and imitation learning. In this research direction, it is proposed to train an auxiliary meta-model that solves the problems of selecting the most informative examples for a human annotator, updating the training hyperparameters of the main model during active learning, as well as augmentation of training samples (for example, upsampling or expanding the training set using self-learning). This direction lies at the junction of classic machine learning and AutoML.
• Better uncertainty estimates for neural network models with a large number of parameters. Uncertainty assessment is an important element in machine learning, which allows, in addition to directly solving the classification or regression problems, assess how much the model is confident in its answer and how much this answer can be trusted. As a rule, the most informative training examples for machine learning models are objects, for which the uncertainty estimates are high. Therefore, such assessments are often used in active learning. Estimating the uncertainty of deep pre-trained neural network models is a complex and little-studied problem that can be solved using various Bayesian methods. In this research direction, it is proposed to adopt recently developed approaches to the estimation of uncertainty to novel natural language processing models.

• Data Science
• Computer Science
• Applied Mathematics
• Statistics

Other eligibility criteria:

• Basic knowledge in Probability, Statistics, Bayesian methods, Natural Language Processing, Deep Learning.
• Programming skills: Python
• High interest in Deep Learning and Natural Language Processing.
• Software engineering skills.
• Previous experience in using Deep Learning is desirable.
• Motivation to complete a PhD within 4 years

As a result, a model trained on data from the web can potentially generate an undesirable text. Since the preemptive check of text appropriateness is impossible, using neural models in commercial applications is too risky. This often makes developers of generation models resort to less flexible, but more reliable and controllable models, such as rule-based dialogue systems based on scenarios.

The goal of this work is to increase the reliability of neural dialogue models by creating an external automatic censor that will evaluate a generated utterance and decide if it is acceptable for a dialogue. The censor will consist of two components: a model that evaluates the quality of generated text and a model that re-formulates the unacceptable text. The Quality evaluation model will define if a text contains inappropriate elements, namely obscene phrases, offense, too informal phrases, undesirable topics. It will also need to model the dialogue flow because a neutral phrase can become offensive in some contexts. The re-formulation model should re-write an unacceptable utterance in such a way that it keeps its content but the tone becomes acceptable.

There currently exist no systems that could identify and correct unacceptable utterances on the fly. The existing automatic censors for chatbots usually rely on lists of potentially dangerous keywords. This approach can overlook offenses which do not contain obscene words. We would like to create a model that considers features of a higher level.

Moreover, existing models do not suggest inoffensive alternatives for the discovered offensive messages. While text paraphrasing and style transfer of texts is an active topic of research, the task of transforming toxic text to neutral has only been considered in a few research works. Thus, the work will include collecting new datasets and devising new methods that take into account the specificity of the task.

• Data Science
• Computer Science
• Applied Mathematics
• Statistics

Other eligibility criteria:

• Basic knowledge of Machine Learning, Natural Language Processing, Probability, Statistics.
• Programming skills: Python, experience with Machine Learning and Neural Network libraries is desirable.
• High interest in NLP.
• Previous experience in the development of Neural Networks for NLP is desirable.
• Motivation to complete a PhD within 4 years.