What is Data Architecture?

Data architecture refers to the organized process of gathering data from various sources and navigating data sets, practices, and platforms from their current state to a desired future state. This is achieved through established frameworks and models, ultimately transforming data into valuable business insights.

In this definition, the term 'description' stands out, highlighting the articulation of the data lifecycle—from collection to processing, storage, usage, and archival.

Big Data must be structured by solution architects and crafted by specialists. A Big Data solution architect is tasked with creating this architectural description, outlining the path from the current to the target state.

Equally important is the term 'manipulation,' which captures the delicate balance of moving and transforming data structures, items, groups, and repositories. Specialists play a crucial role here, depending on project needs.

This encompasses vital architectural and design activities, such as integrating data artifacts into the application landscape, managing communication, interactions, data flow, analysis, source and destination locations, and understanding data consumption profiles.

Understanding Data and Big Data

Data consists of raw facts, information, or statistics collected and stored for various objectives. It can exist in numerous forms, including numbers, text, images, or other representations of information.

Data serves as the foundation for generating insights, supporting informed decision-making, and facilitating various operations within computing and information systems.

Interpreting and analyzing data leads to meaningful information that helps uncover patterns, trends, and relationships—essential for informed decision-making and problem-solving across diverse fields.

Big Data, however, differs from traditional data. Its distinctiveness lies in its volume, velocity, variety, veracity, and value. In essence, Big Data is akin to a treasure hunt for valuable business insights amid a vast sea of data, requiring advanced tools and expertise to navigate effectively.

To clarify, Gartner defines Big Data as data characterized by high volume, speed, and variety—essentially a vast amount of information that necessitates intelligent management for meaningful insights and sound decision-making.

Let's explore the key characteristics that define Big Data:

• Volume: Refers to the sheer size of data—ranging from terabytes to petabytes or even exabytes, with no strict definition of 'large.'
• Velocity: Indicates the speed at which data is generated, such as real-time data from devices like smartphones and IoT gadgets.
• Variety: Highlights that Big Data can take many forms, including structured, semi-structured, and unstructured data types, such as transaction records and multimedia.
• Veracity: Emphasizes the importance of data quality, as maintaining accuracy and reliability can be challenging amid the vast influx of data.
• Value: Represents the ultimate goal of Big Data—transforming chaotic data into valuable insights for businesses through thoughtful strategies and collaboration.

I often jest with friends that Big Data is not for the faint of heart, given the complexity of its myriad data types and the challenge of extracting valuable insights—like solving a puzzle with countless pieces.

While Big Data shares similarities with traditional data, it cannot be managed using the same methods; fresh approaches and tools are necessary to address its unique characteristics.

The process involves collecting data from various sources, storing it, analyzing it, visualizing it, and maintaining oversight throughout. It may sound overwhelming, but the essence of working with Big Data is not merely the volume of data but employing intelligent analytics to unearth valuable insights.

There are several types of analytics: descriptive analytics tells us what's happening now, predictive analytics forecasts future trends, prescriptive analytics suggests potential actions, and diagnostic analytics identifies why something occurred.

Building Blocks of Data: From Concept to Reality

Handling data is akin to baking a cake—layer by layer, we construct our approach. As Big Data architects, we utilize a top-down methodology, dividing it into three critical layers: conceptual, logical, and physical.

The conceptual layer offers a panoramic view, identifying the essential business entities linked to the data. What are the main components in this data ecosystem? This is what we aim to clarify.

Next, the logical layer delves into the specifics, examining how these data entities interact. It connects various aspects of the data landscape, determining what links to what.

Finally, we arrive at the physical layer, where theory becomes reality. This layer focuses on the actual mechanisms and functions that breathe life into data, translating concepts and relationships into tangible, operational forms.

To ensure effective lifecycle management, we must nurture these layers throughout their journey.

Overview of the Big Data Journey

Being a Big Data architect is akin to captaining a ship. Understanding the entire voyage, from start to finish, is crucial. We oversee all stages of the Big Data lifecycle, guiding the technical journey through its various phases.

While our roles may adapt at each stage, the overarching goal is to supervise the entire process. Based on my experience and industry insights, the Big Data lifecycle parallels the traditional data lifecycle but with distinct nuances.

Here's a breakdown of the unique phases we navigate:

1. Foundations: Establishing the groundwork for the entire journey.
2. Acquisition: Gathering data from diverse sources.
3. Preparation: Ready the data for its next phase.
4. Input and Access: Controlling how data enters and who can access it.
5. Processing: Analyzing the data and deriving meaning.
6. Output and Interpretation: Sharing insights drawn from the data.
7. Storage: Securing valuable information in a safe location.
8. Integration: Ensuring seamless connections between all components.
9. Analytics and Visualization: Utilizing tools to comprehend and present data.
10. Consumption: Allowing appropriate users to leverage the data.
11. Retention, Backup, and Archival: Protecting data for the long haul.
12. Destruction: Knowing when to phase out specific data.

These phases may be referred to differently across teams, but their essence remains intact. The Big Data lifecycle is not a one-size-fits-all model; as the field continues to evolve, we draw lessons from traditional data management to tackle the distinctive challenges posed by Big Data.

Consider these phases as guiding stars on your journey, customizing them to meet your organization’s requirements. The key is to remain adaptable and responsive to the ever-changing data landscape.

Phase 1: Foundations — Setting the Stage

Think of Phase 1 as laying the groundwork for a house. In the Big Data realm, Foundations involve understanding the landscape and preparing for the data journey ahead.

This phase entails:

• Identifying the necessary data and its purpose.
• Defining the scope of our data solutions.
• Establishing roles and responsibilities in our data initiative.
• Preparing the technical landscape for action.
• Considering both technical and non-technical factors.
• Understanding data governance within the organization.

This phase serves as a blueprint for our data journey. It requires collaboration from a project manager, the Big Data solution architect, and data specialists to create a comprehensive plan.

Project Managers compile the Project Definition Report (PDR), detailing plans, resources, risks, assumptions, and dependencies. The Big Data Architect provides an essential overview, shaping the foundation of our data-driven project.

In essence, Phase 1 sets the stage for an exciting data adventure, equipping us for the subsequent phases of our Big Data journey.

Phase 2: Data Acquisition — Collecting the Gems

With our foundation in place, it’s time to gather the data treasures for our Big Data expedition. This phase is akin to embarking on a voyage to explore the data seas.

In Data Acquisition:

• Data flows in from various directions—internal and external sources, structured and unstructured formats, including video and audio.
• Data and database administration experts lead the collection of these valuable datasets.
• While specialists handle the heavy lifting, the Big Data architect ensures everything is organized. We are not just bystanders; we lead this phase.
• We establish protocols to ensure data security, privacy, and quality control.

The lead Big Data architect, in collaboration with enterprise and business architects, drives strategy, documenting use cases and detailing technical requirements. Team collaboration is key; domain architects and data specialists join forces, each contributing vital expertise.

Imagine us as captains steering the ship through uncharted data waters. With effective leadership and a strong team, we are prepared to navigate the vast oceans of information in the next phases.

Phase 3: Data Preparation — Refining the Gems

With our data treasures collected, it’s time to polish them for optimal usability. This phase is about cleaning and refining the gems unearthed during our data exploration.

In Data Preparation:

• Raw data undergoes thorough cleansing. Some call it data cleansing, while others prefer preparation. Our goal is to make it shine.
• We act as data detectives, identifying inconsistencies, errors, and duplicates. Anything extraneous is removed.
• Redundant, duplicated, or incorrect data is eliminated, aiming for pristine, usable datasets.
• While data specialists perform the detailed cleaning, the Big Data solution architect oversees the entire process, ensuring alignment with the overarching vision.

Think of us as master jewelers, meticulously refining each piece for maximum brilliance. Our data sets are now polished and ready for the next stages of our data journey.

Phase 4: Data Input and Access — Directing Data to Its Destination

Our polished data gems are prepared to embark on their journey to designated destinations. This crucial phase focuses on directing and accessing our refined data.

In Data Input and Access:

• Data input involves sending our refined data to its intended locations, whether it be a CRM application, a data lake for data scientists, or a data warehouse for specific departments.
• Data specialists take center stage, transforming the cleaned data into a format optimized for its destination.
• Various methods of accessing data come into play, including relational databases, flat files, and NoSQL options, each chosen based on the data's requirements.
• The Big Data solution architect oversees the process, ensuring the team manages the details effectively.

Picture us as air traffic controllers, ensuring every data gem takes off smoothly, following its designated flight path. Our refined data is on the move, ready to create impact in CRM systems, data lakes, and warehouses.

Phase 5: Data Processing — Transforming the Raw

Welcome to the realm of Data Processing, where raw data undergoes a transformative journey. This phase isn’t just about interpreting data; it’s about orchestrating a symphony of insights.

In Data Processing:

• We begin with the raw data, akin to discovering hidden treasure. It is processed into a readable format that conveys both meaning and context, allowing for interpretation using our analytical tools.
• Various tools come into play, including Hadoop MapReduce, Impala, Hive, Pig, Spark SQL, HBase, and Spark Streaming—each contributing to our data processing orchestration.
• This phase entails more than just making data readable; it involves annotation, integration, aggregation, and representation. Annotation labels the data, integration combines diverse data sources, and aggregation compiles data into cohesive datasets.
• Processed data adapts its format based on its intended audience, ensuring it meets consumer needs effectively.
• Advanced processing techniques, such as Spark MLib and Spark GraphX, allow for deeper exploration of our datasets.

While the lead Big Data solution architect orchestrates the process, data specialists, stewards, engineers, and scientists all contribute unique skills to create a harmonious blend of expertise.

Data Processing is a journey where raw data finds its melody, transforming into a symphony of insights ready to engage the audience.

Phase 6: Data Output and Interpretation

In the Data Output and Interpretation phase, data prepares to present its findings to business users. Here, we focus on the essentials:

• Ready for Consumption: Data takes its final form, crafted for easy access by business users. It may appear as plain text, graphs, processed images, or videos—whatever best serves the audience.
• Data Ingestion: This phase also acts as the Data Ingestion process. Once the data is ready, it moves to its storage destination, preparing for its grand entrance into the archives.
• Real-time or Batch: Depending on the context, data can be ingested in real-time or batches, utilizing tools like Sqoop, Flume, and Spark Streaming to manage the influx.
• Interpreting the Act: The crux of this phase lies in interpretation, examining the ingested data, extracting meaningful information, and addressing key questions related to Big Data business solutions.

In summary, Phase 6 is the grand reveal—the moment when data steps into the spotlight, ready to share its insights with the eager audience of business users.

Phase 7: Data Storage — The Secure Vault

As we conclude the Data Output phase, it’s time to securely store our data in purpose-built storage solutions.

In this phase:

• Storage Blueprint: Designated storage units, meticulously outlined in the data platform, are prepared. This blueprint considers vital non-functional factors—capacity, scalability, security, compliance, performance, and availability.
• Storage Infrastructure: These units may consist of Storage Area Networks (SAN), Network-Attached Storage (NAS), or Direct Access Storage (DAS). Data and database administrators manage stored data, controlling access for specified user groups.
• Underlying Technologies: Big Data storage extends beyond basic units; it includes advanced technologies such as database clusters, relational data storage, and specialized systems like HDFS and HBASE, celebrated for their open-source capabilities.
• File Formats Matter: The choice of file formats—text, binary, or specialized formats like Sequence, Avro, and Parquet—plays a critical role in shaping the storage design phase.

Phase 7 ensures our data is securely stored, readily accessible when needed, and prepared for the next chapter of its data journey.

Phase 8: Data Integration — Connecting the Dots

In traditional contexts, storing data often signifies the conclusion. However, in the world of Big Data, another chapter emerges—Data Integration. Here’s the breakdown:

• The Integration Saga: Stored data is not meant to exist in isolation; it seeks interaction with various systems for diverse purposes. Data Integration is a crucial architectural move in the Big Data landscape.
• Architectural Complexity: This process is intricate, as Big Data architects navigate the use of various data connectors, such as ODBC, JDBC, Kafka, DB2, Amazon S3, and more, depending on the data sources involved.
• Connect-the-Dots: Some data models require a harmonious blend, integrating data lakes with data warehouses or marts, while application integration aligns Big Data with dashboards, Tableau, and other visualization tools.
• Overlapping Realms: The integration process may coincide with Data Analytics as they collaborate to reveal insights.

Phase 8 symbolizes the moment where data connections intertwine, setting the stage for the grand symphony of insights to follow.

Phase 9: Data Analytics and Visualization

Within the vast realm of integrated data, a pivotal chapter unfolds—Data Analytics and Visualization. Let’s clarify this phase:

• Integrated data is not merely passive; it demands interpretation and revelation. Data Analytics plays a central role in extracting business value from the Big Data narrative, while Visualization transforms raw data into meaningful insights.
• Equipped with analytical tools, data analytics and visualization come to life. Languages and frameworks like Scala, Python, and R notebooks empower the extraction of insights from the Big Data landscape, with Python often taking center stage for machine learning initiatives.
• In this orchestral arrangement, a chief data scientist may lead the analytics team, ensuring alignment with business objectives.

While Big Data solution architects maintain a supportive role, their vigilance ensures that the lifecycle is executed with architectural integrity—a collaborative dance between data scientists and architects to guarantee that insights are not only uncovered but also architecturally sound.

Phase 10: Data Consumption

After the analytical process, data transforms into consumable information. This is the stage where both internal and external users, including valued customers, engage with the insights.

In this phase:

• The outcome of data analytics is not just insights; it manifests as consumable information. This phase opens the organization’s doors, providing the gleaned knowledge to stakeholders both within and beyond its walls.
• However, this offering is not chaotic; it is governed by policies, rules, regulations, and guidelines. Architectural input becomes crucial in defining the framework that orchestrates the consumption process.
• Imagine data consumption as a well-regulated service. Data governance bodies play a vital role in establishing regulations that oversee how this service is delivered, ensuring responsible consumption.
• Here, the lead Big Data solution architect assumes the role of a conductor, guiding the creation of architectural policies, rules, and guidelines. It’s not just about delivering data; it’s about delivering it responsibly.

Phase 10 marks the transition of data from analytics into a consumable entity—crafted with architectural precision.

Phase 11: Retention, Backup, and Archival

In this phase, the emphasis shifts to protecting critical data. We engage in a strategic choreography of retention, backup, and archival practices—a meticulous dance of safeguarding and compliance.

• Safeguarding critical data is universally acknowledged as essential. Established data backup strategies, techniques, and tools are employed for systematic data protection.
• The Big Data solution architect leads the way, identifying, documenting, and securing approval for strategies that inform retention, backup, and archival decisions—requiring both technical expertise and a comprehensive understanding of business needs.
• While the architect provides the framework, detailed design may be delegated to infrastructure architects, supported by domain specialists in data, databases, storage, and recovery, who construct the protective infrastructure.
• In some cases, data preservation is necessary for regulatory or business reasons, prompting archival decisions. A documented data retention strategy becomes vital, requiring approval from governing bodies, particularly enterprise architects. Implementation is a collaborative effort involving infrastructure architects and storage specialists.

Phase 11 is not just about securing data; it’s about constructing a fortress of protection and compliance—an essential segment in the lifecycle where data durability is meticulously ensured.

Phase 12: Data Destruction — Conclusion and Renewal

In the grand finale of the data lifecycle, destruction takes center stage—a choreographed act driven by regulatory mandates and industry specifics.

Regulatory requirements outline the scheduled disposal of certain data types. Timelines are established, and compliance becomes paramount. This phase is not merely procedural; it is a legal obligation demanding careful attention.

Specific requirements for data destruction can vary significantly across industries. What applies to one sector may not be relevant to another. Thus, adaptability is crucial, allowing the process to align with the unique needs of each organization.

While the data lifecycle suggests a linear progression, real-world scenarios often reveal a more dynamic interplay. Some phases may overlap, creating a nuanced rhythm of operations—synchronized yet flexible.

The lifecycle outlined here serves as a guideline—an awareness of the overarching process. However, it is not a rigid framework. Think of it as sheet music, awaiting a skilled conductor—your data solution team—to bring it to life.

As the curtains close on the lifecycle, the essence of destruction signifies not just an end but a gateway to renewal—an acknowledgment that, in the data realm, every conclusion paves the way for new beginnings.

For further insights on digital transformation, consider exploring the Digital Transformation Handbook for Solution Architects, summarized by Dr. Mehmet Yildiz on Medium.

Thank you for engaging with my perspectives.

If you are intrigued by technology, business, and leadership topics, I encourage you to follow my new account and subscribe to my mailing list on Substack.

Dr. Mehmet Yildiz publishes technology, business, and leadership content from this account.

I have written numerous articles from my main account that may inform and inspire you. Topics include cognitive function, significant health conditions, valuable nutrients, ketogenic lifestyles, self-healing, weight management, writing/reading, and humor.

I publish my health and wellness stories on EUPHORIA. My posts do not include professional or health advice; they document my reviews, observations, experiences, and perspectives to provide information and create awareness.

To receive my posts directly in your inbox, consider subscribing to my content. Writers are welcome to join my publications by sending a request via this link. I support over 24,000 writers contributing to my publications on this platform.