Big Data and Cloud Computing

The sets of large and complex data which the traditional data processing software applications lack behind to handle are generally termed as Big Data. Big Data is mainly used to increase the storage capability and increase the processing power. Similarly, the process where sharing of data and resources takes place with other devices on demand is known as the Cloud Computing. Cloud refers to the internet or network and Cloud Computing refers to accessing, configuring and manipulating the online applications, which offers storage of data, application and infrastructure over internet as a form of cloud. The structure of the report is situating, problem solving and presenting solution evaluation of the business in the organization.

The main purpose is to provide an overview on the various aspects of big data and cloud computing on the emphasis on an organization. In order to analyze and unders6tand the use of the cloud computing and the big data in the decision making in an organization, IBM has been selected as the case study organization. The structure of the report is situating, problem solving and presenting solution evaluation of the business in the organization.

Big Data and Cloud Computing plays a significant role on providing a new business model in this digitalized era. This leads to better decision making. The report is on the impact of cloud computing, Big Data and mobile platform of an organization’s critical evaluation of business. Concentrating on the part where we will discuss about different aspects of Big Data and Cloud Computing.

Big Data is a data set that tends to be in such a large amount that traditional data Processing Software Application lack behind to handle (Marz and Warren 2015). It is more complex and includes challenges like storage, analysis, data capturing, transfer files, sharing, visualization, querying, updating information and privacy (Erl, Khattak and Buhler 2016). Tools that are typically used in Big Data scenarios are:

• NoSQ, which includes DatabaseMongoDB, Riak, BigTable, CouchDB, Cassandra, Redis, Hbase, Hypertable, Voldemort, ZooKeeper.
• MapReduce, includes Hadoop, Hive, Pig, Cafferine, Cascading, mrjob, Cafferine, S4, MapR, Flume, Oozie, Greenplum, Kafka, Acunu.
• Storage, includesS3, Hadoop Distibuted File System
• Servers, includes EC2, Google App Engine, Beanstalk, Herouk, Elastic.
• Processing includes R, Yahoo Pipes, Tinkerpop, Datameer, BigSheets, Mechanical Turk, Solr/Lucene, ElasticSearch (Rahimi et al. 2014).

Business Intelligence is data-driven decision-making, that includes the generation, analysis, aggregation and visualization of data to inform and facilitates business management. Business Intelligence is a new concept, but Big Data is increasing rapidly in IT sectors. Big Data is growing rapidly as the organization commit their resources for trapping the terabytes that flows into their organization and flows out to social media data or sources. Accommodation of advanced analytics for Big Data with Business Intelligence system is a step towards profit of full return on investment (Liebowitz and J 2013). Advanced analytics and Business Intelligence can be highly complementary, advanced analytics can provide a broad exploratory aspect on the data. The Business Intelligence provides users a more structured form of data experience.  The system approaches to some vital analytics and it’s richness in dashboard visualization, performance management matrices, reporting and many more (Breiter et al. 2014). Analytical tools report on historical data focuses on forecasting future events and characteristics, allowing business to conduct and predict the effects of potential changes in business strategies. Analytic focused areas include:

• Predictive analytics
• Prescriptive analytics
• Descriptive analytics
• Geo and temporal intelligence
• Situational awareness
• Advanced data visualization
• Big data analytics
• Cluster analysis

Business Intelligence and Advanced Analytics

Knowledge management in Big Data has three main functionalities Volume, Variety and Velocity. These three dimensions explain the Big Data. Volume though does not have a precise definition but crosses the dimension when a relational database is no longer effective for analyzing the data.  Variety is the amount of information in documents and social media that can work together to sink the structured relational databases and other contents. Velocity is a third factor which has a large amount of data and shall be processed quickly. Velocity can vary that may affect the analytical outcomes (Field, 2013). The ultimate goal of Knowledge Management is to integrate information from multiple perspectives to provide better understanding of decision-making.

In the selected organization, that is IBM now-a-day analyses the email messages, files, videos, distributes and scale out systems within lower budget. IBM Netezza Customer Intelligence Appliance combines some different technologies in a single platform. The user layer relies on IBM Congos® BI software, business intelligence and reporting product. Data warehouse storage layer , Netezza works better for the MPP database system. Use of Hadoop or Cassandra for unstructured and semi-structured data creates an integrated BI and Big Data analytics platform.

Big Data has emerged within the past years as an ideal provider of data and opportunities to enable research and decision-support application, as we have discussed earlier, Big Data faces certain challenges like:

Storage challenges pose the volume, velocity and variety of Big Data. Storing data in earlier days were problematic since traditional storages often fail to store large amount of data. In traditional storage system it is difficult to achieve the velocity of Big Data required in storage systems that is to be scaled up as fast as possible (Kumar et al. 2014). Cloud storage services offers virtual storage which are unlimited and has higher fault tolerance providing potential solution to address the store challenge in Big Data. Transferring and hosting Big Data on the cloud is quiet expensive as compared to other sources (Field 2013).

IBM storage solutions provide users the ready data access with speed and performance with the acuteness of hybrid cloud and software-defined storage. It connects the data across any storage and architecture from IBM. Delivering the intuitions much faster and hence giving the edge of outthinking. Different types of storage in IBM are: Flash storage, Software-defined storage, Hybrid storage, Tape storage, Storage area network and converged infrastructure.

Data transmission process has a life cycle which is: (i) sensors to storage data collection; (ii) data integration from multiple data centers; (iii) data management that transfers the integrated data to cloud platform; (iv) data analysis to moving data from storage for analyzing host. Since data transfer takes place in large amount, challenges in each stages takes place. Therefore, smart preprocessing techniques and data compression algorithm effectively reduces the size of data before transferring.

Knowledge Management in Big Data

IBM uses SSL for data transmission security. All IBM Congos communicate with the Admin Server using SSL. All the files and certificates are required to implement SSL. Setting several parameters is a must (Ibm.com 2017).

The data management paradigm demands for new technologies to clean, store and organize unstructured data. Metadata are effective used to integrate data provenances, but still the challenge remains to automatically generate metadata to describe Big Data and relevant the processes. The variety and veracity of Big Data are used to refine the data management paradigm.  Big Data itself poses challenges to the Database Management System (DBMS) because traditional Relational Database Management System (RDBMS) lacks scalability for managing and storing unstructured data. NoSQL databases are designed for Big Data such that to handle geospatial algorithms which remain another challenge issue (Zakir, Seymour and Berg 2015).

Data Management is critical and turns the organization into hybrid environment to work on multiple types of data and analytics platform. IBM designs to take advantages of builds-once, deploy-anywhere, queries are made simple and increase flexibility and agility on data-storage. There are several Data Management products in IBM like DB2 for Linux, UNIX and Windows (Ibm.com 2017). These are generally used for enhancing the memory performance, making access to memory business data increasing in customer satisfaction, massive stability and flexible the deployment.

Processing a large volume of data needs dedicated resources and hence is partially handled by the speed of CPU, network and storage system. The computer resource is required for processing Big Data. Cloud Computing offers virtually unlimited and no-demand processing power as a partial solution. There are number of issues limiting the cloud ushers. First is in which limitations of cloud computing network bandwidth takes place, impact of on the computation efficiency over large volumes of data (Sharma and Navdeti 2014).

IBM (DPD) Data Processing Division serves the primary sales and marketing of the organization in US. Offering a wide range in variety of Data Processing products. The main target was to work closely to the customers to know and learn their growing needs and developing software/hardware as per the needs.

It is an important aspect in the value chain of Big Data for information mining and predictions. Analyzing Big Data challenges the underlying algorithms in a very complex and scalable way. Sophisticated scalable and interoperable algorithms are required for Big Data analysis and are addressed by Hadoop (Baesens et al. 2014). The existing analytical algorithm requires structured homogeneous data and has difficulties in processing the heterogeneity of Big Data.

Data Storage Management

IBM Db2 saves data up to 80% as compared to any other Database Management system (Ibm.com 2017). Holding the data in higher standards and hence helps to expand data whenever needed, integrating multiple platforms, workloads and dialects.

Big Data visualization uncovers hidden patterns and discovers unknown correlations to improve decision-making (Zakir, Seymour and Berg 2015). The SAS summarizes five key functions for Big Data visualization: (i) highly interactive graphics incorporating data visualization best practices; (ii) integrated, intuitive and approachable visual analytics; (iii) web-based interactive interfaces to preview, filter or sample data prior to visualization; (iv) in-memory processing; (v) easily distributed answers and insight via mobile devices and web portals.

IBM follows several techniques for Data visualization: Personal Touch, increase in quality and types of data, needs some uniqueness by customizing the basics. Deep Engagement  and Embrace Imperfection.

Data integration is critical for the achievement of value of Big Data through integrative data analysis as well as cross domain collaborations. Metadata is required and is essential to track the mappings, such that to make the integrated data sources radically resolvable. However automatically creating metadata is still a challenge (Kwon, Lee and Shin 2014).

Data Integration transforms structured and unstructured data and delivers it to any system on scalable big data platform. IBM data integration helps in understanding, monitoring, transforming, cleansing and delivering data such that to make sure the information given is trust worthy, consistent and judged in real time.

Big data is gradually transforming and hence also affecting the way scientific researchers are conducted. This transformation poses challenges to system architecture. Ideal data architecture would easily synthesize and share data, network, resources, tools, models and people as well.

Data breach and hacking has increased with the increase of dependency on computers and internet, making business and individuals. Big Data poses several new security challenges for traditional data encryption standards, methodologies and algorithms. Data security policies works with the structured data stores in conventional DBMS, which are not effective in handling unstructured data (Lafuente 2015).

IBM serves companies that are data-driven and faces serious threats. IBM security has an integrated system of analytics, real-time defenses and expertise such that to make strategic decisions in business where ever needed. Some of the systems are Security Transformation services which optimize the security strategy and management of the system, Security Operations and response stops threats and finally the Information risk and protection manages the risk as well as protects data in interconnected world.

Data Transmission and Security

The unprecedented networking among smart devices and computing platform contributes to Big Data but poses privacy concerns where an individual’s location, behavior and transactions are digitally recorded (Xu et al. 2014).

IBM helps in establishing policies that governs the organizational way by gathering and managing data, reducing the risk and ensuring global privacy to the system.

Data quality has four aspects: accuracy, redundancy, completeness and consistency. Data accuracy and completeness are made by Big Data as it has the intrinsic nature of complexity and heterogeneity.  Hence, ensuring data consistency and integrity is challenging the Big Data, when the data change frequently and shares with more than one collaboration.

IBM’s data quality product acts on a relied view, accelerate data governance and modernize the system retiring the old database. IBM InfoSphere Information Server is the renowned product.

Cloud computing refers in manipulating, configuring and accessing the application data online, which offers a storage, data infrastructure and application in a form of cloud. Mobile Cloud Computing (MCC) is a the internet rich media experience and requires less processing in cloud, data is stores and mobile devices are serve as a media to display. MCC uses five kinds of resources: Distant mobile cloud, Distant immobile cloud, Proximate mobile computer entities,  Proximate immobile entities and Hybrid (Kumar et al. 2014) 

Cloud computing has made a potential approach toward the attention of business across the IT industry. Capitalizing its speed, scales controls and economic approach. Organizations are relying on cloud marketing, selling, developing products and manage the supply chain around the world (Chang 2015). The cloud workloads creates a hybrid environment by investing for years. IBM interviewed some industrial executives who are staring their transitions by cloud and the feedback, revealing by customers to pay more attention to security, adjusting the business by spending enough time and having an integrated planning. Therefore it is more important for the organization to concentrate on the security, integration and transparency of data which will create a greater value in business.

There are seven key challenges:

• Bandwidth costing.
• Concerns about security.
• Monitoring and supervising continuously.
• Proper Usability.
• Issues in migration.
• Integration and accessing data.
• Cost assessment (Lu et al. 2014).

The IT industry has not only affected by the cloud space but also Big Data as well. Adoption of Hadoop has growing rapidly and the ability of performing analytics on non-corrective and affordable hardware has become more everywhere. IBM Interconnect 2013 has increases value from data insights and gains through big data analytics supported by a cloud infrastructure (Breiter et al. 2014). The benefits of hybrid cloud and big data are harnessed which refers to the explosion in unstructured data. 

Data Management in Big Data

The flow in the volume of data is now a day presenting as a challenge to the cloud servers. Data Architecture are built by organizations and storage policies are also being practiced to work with structured data, where as the unstructured data does not fit the RDBMS framework. This results in manipulation and extraction of data essence in case of simply storing and retrieving the data (Sadiku et al. 2014).

IBM collects and troubleshoots the data collection in two ways. If experienced performance issue, there are two ways to collect system data, first is to take a system snapshot and collect summary of the data and performance state, it helps in building the files in approximately 5 minutes and the second option is to allow collection of historical system information for a time period past up to 12 hours. This helps in collecting data soon after the problem is faced (Inukollu et al. 2014).   

Conclusion

Big Data and Cloud Computing brings new and exciting opportunities to companies who utilizes the available platform. Big Data Hadoop is an open-source software framework that processes and stores Big Data. Whereas Cloud Computing increases probability by improving resources utilization, lower cost and appropriate delivering resources enables organizations to access data correctly. Mobile Cloud Computing still needs to make its architecture neutral, issues regarding MCC are still on wok since users shares personal details over the cloud hence a security issue emerges. IBM (International Business Machine Coorporation) was found in 1911, 16^th June by Charles Ranlett Flint. From a small manufacturer company the company has come a long way researching on various software as well as hardware development(History graph of IBM refers to Appendix). IBM Cloud Computing is exploring new fields for success and development of the industry like medical science, commonwealth banking, security systems, children health care and many more. To conclude this report, in spite of the constant evolution Big Data and Cloud Computing are proved to be an ideal combination where together they can provide a cost effectiveness and scalable infrastructure to support business analysis and IT developments. While Big Data can be tackled by some advanced techniques, Cloud Computing is very important for storage purpose in large industries.

From the above report, we can recommend that the MCC to be more secure and to reduce data breaches from network and security issues to be more strictly handled. And increase the use of Big Data in more specialized fields. 

Data Processing and Cloud Computing

References

Baesens, B., Bapna, R., Marsden, J.R., Vanthienen, J. and Zhao, J.L., 2014. Transformational issues of big data and analytics in networked business. MIS quarterly, 38(2), pp.629-631.

Breiter, G., Behrendt, M., Gupta, M., Moser, S.D., Schulze, R., Sippli, I. and Spatzier, T., 2014. Software defined environments based on TOSCA in IBM cloud implementations. IBM Journal of Research and Development, 58(2/3), pp.9-1.

Chang, V., 2015. A proposed cloud computing business framework. Nova Science Publisher.

Erl, T., Khattak, W. and Buhler, P., 2016. Big data fundamentals: concepts, drivers & techniques. Prentice Hall Press.

Fernández, A., del Río, S., López, V., Bawakid, A., del Jesus, M.J., Benítez, J.M. and Herrera, F., 2014. Big Data with Cloud Computing: an insight on the computing environment, MapReduce, and programming frameworks. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 4(5), pp.380-409.

Field, A., 2013. Discovering statistics using IBM SPSS statistics. Sage.

Ibm.com. 2017. IBM About IBM – United States. [online] Available at: https://www.ibm.com/ibm/us/en/ [Accessed 29 Jul. 2017].

Inukollu, V.N., Arsi, S. and Ravuri, S.R., 2014. Security issues associated with big data in cloud computing. International Journal of Network Security & Its Applications, 6(3), p.45.

Kumar, L., Malik, N., Agghi, G. and Anand, A., 2014. Mobile cloud computing. IJRIT International Journal of Research in Information Technology, 2(9), pp.787-792.

Kwon, O., Lee, N. and Shin, B., 2014. Data quality management, data usage experience and acquisition intention of big data analytics. International Journal of Information Management, 34(3), pp.387-394.

Lafuente, G., 2015. The big data security challenge. Network security, 2015(1), pp.12-14.

Liebowitz, J. ed., 2013. Big data and business analytics. CRC press.

Lu, R., Zhu, H., Liu, X., Liu, J.K. and Shao, J., 2014. Toward efficient and privacy-preserving computing in big data era. IEEE Network, 28(4), pp.46-50.

Marz, N. and Warren, J., 2015. Big Data: Principles and best practices of scalable realtime data systems. Manning Publications Co..

Rahimi, M.R., Ren, J., Liu, C.H., Vasilakos, A.V. and Venkatasubramanian, N., 2014. Mobile cloud computing: A survey, state of art and future directions. Mobile Networks and Applications, 19(2), pp.133-143.

Rittinghouse, J.W. and Ransome, J.F., 2016. Cloud computing: implementation, management, and security. CRC press.

Rittinghouse, J.W. and Ransome, J.F., 2016. Cloud computing: implementation, management, and security. CRC press.

Sadiku, M.N., Musa, S.M. and Momoh, O.D., 2014. Cloud computing: opportunities and challenges. IEEE potentials, 33(1), pp.34-36.

Sharma, P.P. and Navdeti, C.P., 2014. Securing big data hadoop: a review of security issues, threats and solution. Int. J. Comput. Sci. Inf. Technol, 5(2), pp.2126-2131.

Xu, L., Jiang, C., Wang, J., Yuan, J. and Ren, Y., 2014. Information security in big data: privacy and data mining. IEEE Access, 2, pp.1149-1176.

Zakir, J., Seymour, T. and Berg, K., 2015. BIG DATA ANALYTICS. Issues in Information Systems, 16(2).