Statistics is Reborn

The science of statistics is one of the key instruments to derive significant information from our tally data sheets to complicated database system. We could basically explore the relational patterns and validate the hypotheses in scientific experiments or business questions in the past decades, it is yet possible to predict more complicated life events and human behaviors via billions of records. Accordingly, analytical instruments also evolve functionally to analyze the data with the lowest margin of error.

In the process of analytical evolution, new concepts such as data mining, big data and brand-new data ocean are born by defining interdisciplinary sub-fields at the intersection of statistics,artificial intelligence and database systems. Modern definitions for analytical world combine the data scientists from different disciplines as well as to promote the usage of data-driven decisions.

The data size has grown about more than ten times in the past decade, so the challenge for big data analytic gets to be very significant. For this reason, high-tech companies manufacture big data solutions to reduce build-up cost and risk of analytical database systems, which is the very first phase of analytical process. However, statistical design which is the next phase of this process is also significant to get accurate decisions.

Big data helps to increase both the size (i.e. number of rows) and the complexity (i.e. number of variables, tables or relations) of data. From the statistical viewpoint, number of non-missing rows indicates the sample size of our data which is directly proportional with the measure of statistical significance (i.e. log worth) for descriptive data indicators. Therefore, statistics generated by big data usually does not need to be tested via statistical hypothesis because of very low standard error due to huge sample size. For uni-variate analysis, it seems that we do not need to use statistical procedures to inference the big data. However, we must need to use many related variables together to construct statistical data indicators or models. For example, predictive modeling is the essence of analytic to turn large amounts of data into easy-to-use mathematical models. In this context, best mathematical model and most significant set of predictors can only be discovered from the complex data by the help of statistical variable selection, model fitting and evaluation algorithms. Therefore, the usage of statistical science and probability theory increases heavily with each passing day.

Recently, complex data sets are frequently simplified by predictive models to predict life events and human behaviors. On the other hand, business intelligence (BI) dashboards are also utilized as a data visualization and summary tool for big data. Furthermore, the ability to real time data from multiple sources, a customizable interface and filtering functions of BI dashboards enable us to examine the information down to the lowest level of detail, for immediate and in-depth analyses. However, the results from these tools are still too complicated for end-users because of insufficient statistical structure. If the science of statistics aims to simplify and highlight the significant data as in predictive modeling, new innovations are still needed in terms of statistical design for these analytical tools.

Analytics’ our passion.