About

I am a Machine Learning Research Scientist at Apple.
I got my PhD degree from the University of Waterloo, where I worked with Ming Li on language modeling and unsupervised machine learning methods.
Before that, I worked with Chengqing Zong on spoken language understanding during my master’s study.

In general, my research investigates how to represent language for computing. Lately, I am obsessed with language modeling which represents language via neural computing for its unsupervised and task-agnostic nature. I am also interested in multilingual problems and acoustic sequence modeling.

My past works concern modeling text and speech sequences to achieve lower perplexity, better generation, and benefit downstream language tasks; specifically, we address the problem of modeling text and text-speech sequences with Transformer-based language models. My favorite works during my Ph.D. study are Segment-Aware Language Modeling, Hypernym-Instructed Language Modeling, and Alignment-Aware Acoustic and Text Modeling.

Publications

Xiaoran Fan, Chao Pang, Tian Yuan, He Bai, Renjie Zheng, Pengfei Zhu, Shuohuan Wang, Junkun Chen, Zeyu Chen, Liang Huang, Yu Sun, Hua Wu. ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech. (preprint) [pdf][code]

Peng Shi, Rui Zhang, He Bai, Jimmy Lin. XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing. Finding of EMNLP 2022. [pdf][code]

He Bai, Renjie Zheng, Junkun Chen, Xintong Li, Mingbo Ma, Liang Huang. A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing. ICML 2022 (full paper) [pdf][code].

He Bai, Tong Wang, Alessandro Sordoni, Peng Shi. Better Language Model with Hypernym Class Prediction. ACL 2022 (full paper) [pdf] [code].

Peng Shi, Rui Zhang, He Bai, Jimmy Lin. Cross-Lingual Training with Dense Retrieval for Document Retrieval. EMNLP-MSR 2021 (workshop paper) [pdf].

He Bai, Peng Shi, Jimmy Lin, Luchen Tan, Kun Xiong, Wen Gao, Jie Liu, Ming Li. Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation. ACL-SRW 2021 (workshop paper) [pdf] [code].

He Bai, Peng Shi, Jimmy Lin, Yuqing Xie, Luchen Tan, Kun Xiong, Wen Gao, Ming Li. Segatron: Segment-awareTransformer for Language Modeling and Understanding. AAAI 2021. (full paper) [pdf] [code]

Peng Shi, He Bai, Jimmy Lin. Cross-Lingual Training of Neural Models for Document Ranking. EMNLP Findings 2020. (short paper) [pdf] [code]

He Bai, Yu Zhou, Jiajun Zhang and Chengqing Zong. Memory Consolidation for Contextual Spoken Language Understanding with Dialogue Logistic Inference. ACL 2019. (short paper) [pdf] [code]

He Bai, Yu Zhou, Jiajun Zhang, Liang Zhao, Mei-Yuh Hwang and Chengqing Zong. Source Critical Reinforcement Learning for Transferring Spoken Language Understanding to a New Language. COLING 2018. (full paper) [pdf]