The biggest idea that has captured the attention of the IT industry today is “Big Data.” One of the main reasons why the “Big Data” field is so popular is that new technology platforms provide the capability to process data of multiple complexities, without the constraints faced by earlier technology platforms.
Before we explore further, let’s examine “what is Big Data?” Big Data can be defined as a collection of data sets so large and complex that it is difficult to process using traditional database management tools, data processing methods or other technologies available in the marketplace. (Running data beyond an exabyte—1 billion gigabytes—would prove too slow).
The most common question technology professionals ask today is “why Big Data now?” The important thing to understand here is the data part of Big Data was always present and used in a manual fashion, with a lot of human processing and enhancement, eventually being used in a decision-making process. What has changed and created the buzz with Big Data is the automated data processing capability that is extremely quick, and offers flexible processing.
The promise of Big Data is the ability to access large volumes of diverse data that can be useful in gaining critical insights. This is done from processing repeated or unique patterns of data. The availability of new data processing frameworks and platforms resulted in significantly lower costs and higher scalability than earlier technology platforms.
Applying Big Data to Various Industries
While each organisation will have its own set of data requirements for Big Data processing, here are some examples for vertical markets:
- Healthcare: Pharmaceutical companies can minimise the life cycle of processing data for clinical trials—and manage the same—with rules-based processing using big data platforms. In addition, inputting patient symptom data given by doctors can provide lot of insights into proactive diagnosis.
- Investment Banking / Mortgage Banking/ Accounting / Finance: Corporate performance reports and annual filing to regulatory firms is the most important requirement for all firms but extremely stringent for financial firms. In addition, compliance is another area that could improve using Big Data platforms.
- Automotive Industry / Real Estate/ Logistics / Manufacturing / Utilities: A common challenge encountered by companies in these verticals is maintaining records from facilities, machines, non-computer-related systems, and many more. In addition, different types of labour needs vary according to business cycles, bringing another set of problems that organisations need to solve.
In addition, different organisations execute varying types of contracts every year and there are multiple challenges and legal hurdles associated with each of them.
All industries have their own specific reasons for using big data solutions. However, a similarity shared by all industries is the need to manage and analyse data from various resources. How records are used and the content contained in these information repositories all vary, but they all share the reasoning that data of multiple complexities should be processed from logical sources and then used in a decision-making process.
Author: Amar Naik
Amar is an Application Developer and Project Manager in a wide variety of financial business applications. Was part of major migration of content management system for a investment banking firm. Particularly interested in designing applications right from the scratch. Always interested in working on migration projects, content management and data warehousing applications. Specialties: Datastage, Content Manager On Demand, Unix, Mainframe, Project Management.
View Linkedin Profile->
Other Articles by Amar->