Big data analytics is a current title given to the process of analyzing large amounts of recorded information, for the purpose of discovering valuable insights from the data. It is widely used in the business world as well as in scientific research. The methodologies used for analysis combine statistical models or non-probabilistic models with iterative learning algorithms. The output would typically be a predictive rule that is based on information from many features, and can help anticipating an outcome for newly coming observations. It could also be an identification of a pattern related to the observations in the data, such as clusters or networks.
My talk will focus on several important topics from the statistics perspective that are encountered when analyzing large-scale datasets, and the means of using them in the context of a machine learning process. I will discuss the implementation of a learning algorithm and evaluation of model performance while controlling the statistical error, and demonstrate the methodologies within an analysis performed as part of a scheduling project in industry.
Anat Reiner-Benaim, Department of Statistics, University of Haifa