The dynamic business scenario with trends of liberalization and globalisation is posing new challenges before the management at multiple levels of complexity. Some of the challenges which were not considered seriously in the past, but now play an important role relate to free flow of information, technological changes, organizational restructuring, changing ultimate users needs, increasing competitiveness, etc. The advent of IT has lead to a stage that every organization, be it big or small, has over the years compiled, collected and is in possession of a large volume of data. The organizations, which utilize this information bank for efficient running of the system, will certainly sustain and come up further. For this purpose, the organizations have to find out and adopt a right methodology for better utilization of the information base. Apparently, there is a need to make the system more efficient and cost effective than ever before. The availability of the basic data for information right from the point of generation, at all levels for all the time, is still a problem in most of the organizations. Though the development of computer technologies like Internet and Intranet has further enhanced the utility of these databases as the main supportive system for planning and decision-making, there is still a vacuum in the analysis based decision systems. The growth of IT and its adaptation over the years has been exponential, while at the same time the cost of both hardware and software are decreasing tremendously. New applications in all areas of planning and strategic operation are being developed and used at all levels. Theoretical developments and state of the art tools incorporating the latest and efficient algorithms are also available now, may be not in an integrated fashion. The management applications of the versatile and proven quantitative, management and simulation techniques have a strong development base. The adaptation and applications of these as a system, integrated with ICT of today in the decision-making as a process is still far from reality.
| Present Scenario The growth in IT follows double exponential with no upper limits. Even small enterprises have adopted IT as a part of its growth plan, mainly because of the low cost in developing the infrastructure with significant returns. Moreover, changing IT scenario is also easily accessible even to the new entrants. The processing of the structural information with need based objective(s) for planning and decision-making is required to have an edge. In the government sector, IT is playing a lead role in almost every level of planning with a strong IT support already in place.
| RDBM systems Over a period of time large number of RDBMS have been developed, and are operational within the various central and state government ministries/departments as well as districts of all the states. There is a regular mechanism for maintaining these RDBMS and lot of reports generated for the various kinds of planning and decision making activities in the entire government. Most of the RDBMS are part of the NICNET based management system and thus there is a transportability of these various RDBMS across the entire government for various kinds of reports and applications generation. Thus, the infrastructure, in terms of the computer communication link i.e. the hardware as well as software is in position to integrate the objective based RDBMS for specialized applications. These systems are around heterogeneous software and hardware platforms.
| Business Intelligence Technology The need to improve the decision making capabilities using the ever increasing computing power, availability of RDBMS across heterogeneous platforms led to the use of more and more information in the decision making process. Though the information base in each of the sectors has grown into hundreds of thousands of GBs, the peculiarity of database structure does not allow one to perform a detailed analysis on the data in a way one wants to do. Further, even if one does the analysis on large volume of data one should know the complete designing of the data model and its contents. It is practically not possible for any one to know the complete data modelling or the contents and ad hoc query analysis is simply not possible. This implies that only an expert in data modelling can do any sort of desired analysis. Even then it will result in large number of joints and thus drastically reduce the performance. DSS is being used at all levels within an organizational set up and the organizations are now looking for a frame work to accomplish multiple goals. This is the time the analytical intelligence is needed the most. A simple transaction tells what is happening or happened on-line? Other unknown half is what is going to happen? In order to overcome these shortcomings in the RDBMS, a strong need was felt that a new technology or tool is necessary to support the decision making environment. The new technology should facilitate the decision maker, to the maximum extent, in the decision making process. Moreover, the technology should help in transforming the raw data into useful information and then into a knowledge base for the organization. The transformation process is possible with the adoption of data warehouse technology as depicted in fig-1.
| Data Warehouse, OLAP & Data Mining What is data warehouse? BI/Data warehouse is an essential information technology component for organizations with performance improvement initiatives. At its core, it is the extraction and transformation of data from disparate internal and external sources to information that is used to make decisions in the organization. Data warehouse is a collection of transaction data specifically structured and maintained separately for query and analysis in support of decision-making. In contrast to the OLTP systems, which are built around applications, data warehouses are built around subjects or departments. Moreover, the OLTP model design structure is not suited for analytical applications. If one wants to do analytical applications on the OLTP database itself, it will require lot of joins which in turn will reduce the performance of both the applications. The BI applications are specially meant for decision support, naturally for the top-level management of the organization.
| On-Line Analytical Processing (OLAP) On-Line Analytical Processing (OLAP) is an efficient technology for performing complex analysis from a business perspective while hiding the complexity of underlying data structures. It typically involves analysis of trends and comparisons across business dimensions such as location (state, district, city), time (year, quarter, month), taxes (commercial tax, sales tax, central excise) via analytical operations such as data consolidation, drill-down, and slicing and dicing, OLAP tools allow the user to analyse complex data relationships quickly and easily using historical, projected, and derived data to provide detailed reports. With OLAP tools, users can analyse and navigate through data to discover trends, spot exceptions, and get underlying details to better understand the on-going process. The two forms of analysis most relevant in this context are commonly known as “slice and dice” and “drill-down”.
| Data Mining The data mining technology is like extracting gold and is parallel to gold extraction technology. Data mining is based on filtration of ‘a mountain of data “ore”’, in order to get data “nuggets” and is designed to help organization(s) to discover hidden patterns and to delve deeper to establish hidden connections in organization’s data – patterns that can help planners and decision makers to understand the behavior of key users, detect likely trends/ growth pattern, predict change(s) in the financial position etc. Data mining helps in managing the business effectively and to gain competitive edge.
| Text Mining Organisations generate, collect and have large volumes of data, which they use in day-to-day operations. These data are mostly in the form of numeric and text. The text-based databases are in various forms, like: emails, Parliament proceedings, Technical/Expert group reports, details about patients and treatments, and court judgements. Many of the organisations are unable to capitalise fully on the value of text data because information implicit in the data is not easy to discuss. The tools used to analyse numeric data cannot be put in to use directly for analysing the text data. The need for tools to deal with such databases are already at large. This implies an opportunity to make more effective use of repositories of business communications, and other unstructured data.
| Infrastructure For any BI system to be successful and sustainable, there is always a need for dedicated network services within the organization and outside for Internet access. As NICNET is operational for long and is stable and being upgraded regularly with newer technologies, the basic need for developing BI applications in the government environment is already in place. The RDBMS and other forms of systems are operational in all major sectors and are being upgraded on a regular basis till the district levels. As the BI solutions and analysis reports will be ultimately used by the top level decision makers in the government, the software and hardware platform on which the BI systems are to be implemented should be capable of handling various issues related to the data warehouse and capable of providing web based solutions. The development of BI applications in the government needs the above basic infrastructure and NIC has already acquired the state-of-the-art BI tools which are operational at various locations within NIC. In the government domain there are large number of databases being used for various applications and analysis. Also those applications require deriving analytical intelligence objectively within a limited time frame. As a service provider NIC is fully aware of the gap between availability of information and readiness of supplying the information to the decision makers for decision-making at the right time.
| Hardware and Software Today, business intelligence tools access, analyze, and deliver information in the form of metrics to business users through applications like dashboards, reports, or portals. Complete business intelligence solutions have evolved to facilitate collaborative decision-making that fits the needs of business users in various departments and at various levels. BI technology has become a highly integrated and visible component of both the IT infrastructure and business operations even in the government departments. After assessing NIC’s existing infrastructure as well as the future needs, state-of-the-art tools covering the major processing areas of Analytical/Business Intelligence viz. ETL, OLAP and Data Mining have been acquired and made available on a client server environment. As a temporary measure, the SAN server at NIC headquarters is being used as BI server as well as production server. The BI division is in the process of acquiring two rack mounted servers and soon the BI tools would be migrated to the new dedicated servers. The server version of the tool is installed on the SAN server machine and the client versions on selective client machines at various locations at the center and some of the identified state units. The architectural details are given at fig-2. Presently 20 clients are authorized to connect to the server besides five for data mining and can use the facilities.
| Analytical Intelligence in the Government Over the years the various application divisions at NIC have compiled and generated large volume of data. This data has been regularly processed and reports are generated as per the needs of the policy planners and decision makers. A large number of application/OLTP systems are running in most of the central and state government ministries and departments of which some of the OLTP applications and the related databases might play a key role in the policy planning issues pertaining to a sector. Further, as the government departments generally work with less resources and more demand for services, it may be that different departments might work in isolation with almost similar or same set of priorities and goals and spending more to achieve a common objective. In these circumstances, analytical applications can play an important role in the efficient usage of limited resources and offering maximum services to the user. As mentioned earlier, this is possible only by creating a data warehouse and adopting data warehouse technology that can play a leading role in giving analytical support to the various levels of planners in the government hierarchy. Data warehouse and Data mining is a perfect means of preparing the government to face the challenges of the next millennium.
| Strengthening the Decision Making Process In addition to the OLTP applications running at all the levels, analytical intelligence system can be put in at the middle and higher levels so that the analytical information systems help them in the better management of the organization system, its control and monitoring. In order to strengthen the various NIC divisions in this direction, three sub-groups have been formed to explore the possibilities of undertaking BI projects at various divisions in the center and select state units. In the first phase these groups will look after the center, Kerala, Karnataka, Tamil Nadu, Maharashtra, Madhya Pradesh, Gujarat and Uttar Pradesh. The remaining states will be covered in the second phase. Further, these groups will act as a guiding force in identifying new projects, in resolving issues arising out of these and help in planning and execution of BI projects in the center and state government departments.
| Training of NIC Official The skill upgrade of the existing NIC professionals has been done at two levels: at the senior level officials and the other for the working level officials. Senior officers have been exposed on all major aspects of analytical intelligence by in-depth exposure on the above-procured tools in a client server environment. Working level officials have been trained in-depth covering basic BI concepts and of the tools and its potentials through live case studies. In addition to the training provided by this division as in the past, have also trained other divisions in building the warehouse. These divisions in turn will train other division officials in undertaking development of data marts or data warehouses for the applications identified in their area of operation.
| Technology Demonstrations In order to create awareness and to propagate the technology and its advantage, this division is demonstrating at various platforms, a number of data warehouses developed jointly with the application divisions. Besides above in order to demonstrate the technology to the end user viz. centre and state government officials a number of demonstrations have been conducted mainly to demonstrate the BI potentials and its requirement in the government environment. Some of the typical data warehouse developments going on in other divisions and state units are: Agriculture Informatics Division, Land Records Division, and Water Resources Division –all at the head quarter, Maharashtra, Tamil Nadu and Kerala.
| Applications Though data warehouse applications are not restricted to any particular domain, generally wherever the information has any economic value or influences the financial or economical aspects of the organization, data warehouse can easily fit in to it. Further, any division having large volume of data of either numerical or text or spatial types can go for data warehouse application. The only issue one has to keep in mind is that the department should have sufficient amount of information in their possession with high utility value. Some of the projects taken up in the division include, data marts on – Customs Import, Bhoomi, Socio-economic indicators of UP state, Fertilizer production and consumption, and Export-Import of selective commodities.
| Future Scenario NIC is on its way to support the government on BI technology so as to make it more powerful in terms of strengthening the decision making process on a near real time basis. All RDBMS systems around NICNET will be brought under the BI technology for the end user to view the multi dimensional perspective of the data as well as the likely scenario for the future. This will improve substantially the power of decision making at all levels in the government hierarchy. Besides, BI systems will lead to feasible planning and input from all levels in the planning and decision-making hierarchy will improve efficiency in the government sector.
For further details, please mail to lenin@nic.in