Unit 3: System Documentation Strategy (8 Hrs)

3.1. Quality System and Engineering Data Records

3.2. System Design and Development Data
3.3. Data Accession List (DAL) and Data Criteria List (DCL)
3.4. SE and Development Documentation Sequencing
3.5. Documentation Levels of Formality
3.6. Export Control of Sensitive Data and Technology
3.7. System Documentation Issues
=====================================================================================

Unit 3: System Documentation Strategy

parts of a software project. However, a lot of projects

co-founder of Write the Docs.

51% of people prefer to receive technical support through a knowledge base, and yet producing
the relevant technical documentation can be a headache for many companies.

Luckily, there are many examples of software brands leading the way with documentation, and
s in this post.

Looking at other software documentation examples can inspire your own project, although your
processes will be entirely your own.

Software Documentation: What it Is

Software documentation is a type of documentation that provides information about software

products and systems. It typically includes a wide range of documents and materials that describe
the features, capabilities, and use of the software.

Software documentation can be organized into different categories, depending on the intended
audience and purpose of the documentation. Some common types of software documentation
include user documentation, which provides information that is useful for users of the software;
technical documentation, which provides detailed information about the technical aspects of the
software; and process documentation, which describes the steps and procedures that are used to
develop, test, and maintain the software.

Types of Software Documentation

Software documentation can be broken down into several different categories or types. The types
of documentation that you should create for a software system will depend on the audience and
the intended use of the software. In general, it is a good idea to create documentation that provides
all of the information that users need to effectively use and maintain the software.

For end users, it is often useful to provide user manuals that provide step-by-step instructions for
common tasks and that describe the features and capabilities of the software. It is also often helpful
to provide tutorials or other types of training materials that can help users learn how to use the
software.

For developers and other technical stakeholders, it is often useful to provide reference manuals
that provide detailed technical information about the software, such as its API, data structures, and
algorithms. It is also often helpful to provide process documentation that describes the processes
and procedures that are used to develop, test, and maintain the software.

For system administrators and other IT professionals, it is often useful to provide installation
guides that provide instructions for installing and setting up the software on different types of
systems. It is also often helpful to provide system documentation that describes the hardware and
software components that make up the system, as well as the interactions between those
components.

The key to remember is that each documentation type requires a slightly different approach since
they are aimed at different audiences.

1. Project Documentation

Project documentation typically refers to the documentation that is created during the development
process for a software project. Project documentation is typically intended for use by the
development team and other stakeholders, rather than for end users of the software.

Some examples include:

Technical design documents

Project plans
Project requirements specifications

2. Product Documentation

Product documentation is typically used to refer to the documentation that is created for a specific
software product. This type of documentation is intended to help users understand and use the
software effectively.

Some examples include:

Instructional manuals
Reference manuals
Installation guides

3. Process Documentation

Process documentation is important for software documentation because it provides information

about the processes and procedures that are used to develop, test, and maintain the software being
created.
This information can be useful because it can help software developers and other technical
stakeholders understand the steps that are involved in the software development process, and it can
provide guidance on how to follow those steps. Additionally, it can help ensure that the software
development process is consistent and repeatable, and it can provide a record of the decisions and
actions that were taken during the development process.

Examples of process documentation:

Development plans
Testing plans
Release plans
Bug tracking reports

4. Technical Documentation

Technical documentation is a type of documentation that provides detailed information about the
technical aspects of a product or system. In the context of software documentation, technical
documentation typically provides information about the technical characteristics and capabilities
of the software such as the software's architecture, data structures, algorithms, and other technical
details.

Technical documentation is important for software documentation because it provides detailed

information about how the software works and what it can do. This type of documentation is
typically created to help developers and other technical stakeholders understand the technical
details of the software, and it can provide guidance on how to use the software effectively.
Additionally, technical documentation can also be useful for end users of the software, as it can
provide information about the features and capabilities of the software, and it can help them
understand how to use the software to achieve their goals.

Some examples of technical documentation include:

API documentation - Reference documentation regarding making API calls and classes
Data model documentation - Information about the data structures and relationships that
are used by the software such as the entities, attributes, and relationships that are defined
in the data model as well as examples of how the data model is used by the software.
Architecture documentation - Overview of the overall design and structure of the
software
User guide - Document that provides instructions on how to use the software
Release notes - Information describing the latest changes and improvements in a software
or feature releas as well as any bug fixes
README - A high-level overview of the software, usually alongside the source code

5. System Documentation
System documentation is a type of software documentation that provides information about the
architecture, components, and design of a software system. It is an important type of
documentation because it provides valuable insights into how the software works, and it can help
developers, administrators, and other technical stakeholders understand the system and its
capabilities.

The main purpose of system documentation is to provide technical information about the software
system. This can include things like the system architecture, the components and modules that
make up the system, and the design principles and patterns that were used to build the system. By
providing this information, system documentation can help developers and other technical
stakeholders understand how the system is organized, how it works, and how it can be extended
or modified.

Additionally, system documentation can provide useful information for users who want to learn
more about the system, or who want to understand its capabilities and limitations. For example,
system documentation can provide details about the system's features and capabilities, as well as
information about how to use the system and troubleshoot common issues.

Some examples include:

Troubleshooting guide - provides information that is useful for users who want to
troubleshoot common issues or problems with the system
Architecture documentation - provides information about the architecture of the system,
including the components and modules that make up the system, and the relationships
between them. (Architecture documentation can also be part of any technical
documentation created as we mentioned above).
User manual - Iseful for users who want to learn how to use the system. It can include
step-by-step instructions and examples, as well as information about the system's features,
capabilities, and limitations.

6. User Documentation

User documentation, as the name suggests, is focused on providing information that is useful to
end users of the software. User documentation is intended to help users understand how to use the
software, and it is typically written in a clear, concise, and easy-to-understand style.

Some examples include:

How-to guides Problem-oriented, take the user through a series of steps to reach a real-
world goal
Tutorials Learning-oriented, take the user through a series of steps to learn a concept
Reference docs Information-oriented, technical descriptions of the software (could
include software design documents)
 Explanations Understanding-oriented, they clarify or illuminate a particular topic for a
user

Benefits of Creating Software Documentation

Creating software documentation can provide a number of benefits. Some of the key benefits of
creating software documentation include:

1. Improved User Experience

Software documentation can help users understand how to use the software, and it can provide
information that users need to achieve their goals. This can improve the overall user experience of
the software, and it can help users get the most value out of the software.

2. Enhanced Collaboration
Software documentation can help developers and other technical stakeholders understand the
technical aspects of the software, and it can provide information that they need to work on the
software. This can enhance collaboration among team members, and it can help ensure that
everyone is working towards the same goals.

3. Increased Efficiency
Software documentation can provide clear, consistent, and up-to-date information about the
software, and this can help developers and other technical stakeholders work more efficiently. For
example, developers can use the documentation to quickly find the information they need, and
they can avoid having to spend time trying to reverse-engineer the code or figure out how the
software works.

4. Improved Quality
Software documentation can help ensure that the software development process is consistent and
repeatable, and it can provide a record of the decisions and actions that were taken during the
development process. This can help improve the overall quality of the software, and it can help
prevent errors and mistakes.

How to Write Effective Software Documentation

There are several steps you should take if you want to create software documentation that is
accurate, useful, and easy to understand.

Here are the following best practices to follow when looking to write software documentation.

1. Prioritize Documentation in the Development Process

This may seem obvious, but as we mentioned earlier, software documentation may fall under the
radar due to developers not seeing the value of documentation or not having enough time or
expertise to create high-quality documentation. Additionally, because some organizations may not
have established processes or guidelines for creating and maintaining software documentation, it
can make it challenging for developers to create and update the documentation.

This is why it's the first step to writing effective software documentation is to prioritize it during
the software development lifecycle!

ature unless it is accompanied by the appropriate

documentation.

Hire technical writers who can promote the value of documentation within your company.

Invest in the right tools to make it easy for your development team to create the necessary
documentation.

Whatever the case, it's important that you get everyone on the same page and explain the benefits
of creating software documentation. By understanding the value of software documentation,
developers and other technical stakeholders can make informed decisions about how to prioritize
it in the development process.

2. Identify Your Target Audience

It is important to identify your target audience when creating software documentation because its
your readers who will determine the content and style of the documentation. Different audiences
will have different needs and expectations when it comes to software documentation, and it is
important to understand those needs and expectations in order to create effective documentation.

For example, if your audience for the documents you're looking to write is end users of the
software, the documentation should be written in a clear and concise style, and it should provide
step-by-step instructions for common tasks. It should also provide information about the features
and capabilities of the software, and it should include examples and exercises to help users learn
how to use the software.

On the other hand, if your audience for the software documentation you're creating are developers
or other technical stakeholders, the documentation should provide detailed technical information
about the software, such as its API, data structures, and algorithms. It should also describe the
processes and procedures that are used to develop, test, and maintain the software.

The Splunk Documentation Team provides an in-depth guide in their book The Product is the Docs
on how to define your audience for technical writers.

Splunk, p.275
This is an exercise that is useful not just for technical writers but also for other members of your
company, including marketing, engineering, product, and support.

Define your user. You can start with available user information and talk to customer-facing
teams like support.

Create audience personas.

Create audience definitions (e.g., entry-level admin audience).
Create use cases for the product (e.g., manage enterprise customers in a CRM system).
Identify the correct delivery formats for your users (e.g., FAQ, wiki, or knowledge base).
Create content that is an appropriate scope and at the right level of detail.
Identify appropriate users to provide feedback on your documentation.
Conduct user research and communicate with users.

Remember, your software users may change over time. Repeat this exercise at least once a year.

3. Define the Scope and Goals

Once you have identified the audience, the next step is to define the scope and goals of the
documentation. This can help you focus on the most important information and ensure that the
documentation is relevant and useful. For example, you may want to focus on specific features or
use cases, or you may want to provide information that is needed to complete specific tasks.

4. Develop a Content Strategy

The next step is to plan how you will go about actually creating the necessary software
documentation to meet the scope and goals of the previous step as well as who will be responsible
for maintaining the documentation. This can involve establishing a schedule for creating and
updating the documentation, as well as identifying the tools and resources that will be needed. The
plan can also include a process for reviewing and revising the documentation, to ensure that it is
accurate and up-to-date..

A documentation content strategy helps you keep on track, allocate resources, and manage your
time. It will help you time your documentation alongside releases so you can have some idea of

5. Create a Style Guide

Just like you would create a style guide for your content marketing activities, you should consider
having a style guide for your software documentation.

Having a style guide can be helpful for a number of reasons.

It can help ensure consistency in the software documentation process. By following a set
of standardized rules and guidelines, writers can avoid using conflicting or inconsistent
styles, which can make the documentation more difficult to read and understand.
 A style guide can help establish a clear and coherent tone for the documentation you write.
By using a consistent style and tone, writers can make the documentation more engaging
and easier to read.
It can help improve the overall quality of any software documentation you create. By
following a set of standardized rules and guidelines, writers can avoid common errors and
mistakes, and they can create documentation that is more accurate and useful.
Some things to consider including in your software documentation's style guide include:

Standardized terminology (how to refer to your company and software)

Voice & tone
Page formatting (use of headers, paragraphs, lists)
Guide on word choice (should it be internet or Internet obviously the former!)
Use of visuals and video

You can refer to some well-known software style guides like the Rackspace Style Guide
or the Microsoft Style Guide.

6. Write Clearly and Concisely

You've most likely come across the acronym KISS (Keep It Simple, Stupid). This principle should
be applied when writing your software documentation.

When writing the documentation, it is important to focus on clarity, conciseness, and organization.
Use clear, simple language, and avoid using jargon or technical terms unless they are necessary.
Use headings, subheadings, and other formatting techniques to organize the documentation, and
provide examples and images to help illustrate key concepts.

7. Review and Revise

Your customers should not be the first testers of your documentation, or you have failed to provide
working docs.

This is why documentation should not be published until it has been subject to technical
verification, which is the point where you should review your documentation to ensure that it's
accurate as well as up-to-date and revise as needed.

To review and revise software documentation, it is important to involve other stakeholders who
can provide valuable feedback and suggestions. This can include developers who are familiar with
the software, as well as users who can provide insights into how the documentation can be
improved.

Gathering feedback from users who have actually used the software can help identify gaps or errors
in the documentation, and it can help improve the overall quality of the documentation. Have
anyone who reviews the documentation check it for factual errors, ensuring that it is consistent
with the current version of the software, and verifying that it covers all of the important features
and use cases. You'll then want to consider incorporating any of the feedback or suggestions you
receive to update the documentation as appropriate.

Once the documentation has been reviewed and revised, it is important to continue to review and
update the documentation on a regular basis. This can help ensure that the documentation remains
accurate and useful, even as the software evolves and changes over time.

Additional Best Practices for Writing Software Documentation

You will need to consider the user experience (UX) of your documentation, especially customer-
facing help content.

Just writing the content is half the battle how do end users feel when they actually read your
documentation? Does it help them to achieve the goal they set out to achieve when reading your
documentation? Are they able to easily find what they need?

Take a look at the following best practices for ensuring that your software documentation is useful
and actually read by your target audience.

1. Keep the Documentation Up-To-Date

Software systems are constantly changing, and the documentation should be updated to reflect
those changes. This will help ensure that the documentation remains accurate and useful.

2. Make Use of Visuals

Visual aids, such as images, diagrams, and videos, can be an effective way to illustrate concepts
and ideas, and they can help make your software documentation more engaging and easier to
understand which is particularly helpful for new users who are learning how to use your software.

3. Include Examples and Exercises

Providing examples and exercises can help users learn how to use the software more effectively.

4. Use a Consistent Structure and Format

This will help users find the information they need, and it will make the documentation easier to
read and understand.

5. Consider Hiring Professional Technical Writers

ng wrong with your developers writing documentation if necessary, but it the quality
of your documentation might greatly improve if you make use of professional technical writers.
Technical writers are professionals who specialize in creating clear, concise, and accurate
documentation for technical products and systems. They have the knowledge, skills, and
experience to create high-quality documentation that is useful, easy to understand, and up-to-date.

You can outsource your technical writing needs to a BPO vendor or hire an in-house expert to
create technical documentation for your products or systems.
6. Inclusivity and Accessibility
When you get to a certain point in your documentation, you need to seriously consider how people
with different needs will be able to use your documentation.

For example, consider whether your users are from international audiences when actually writing
content. You want to avoid the use of idioms and references that they might not understand.

Accessibility relates to the User Experience of the documentation tool itself. For example, consider
whether it will be accessible to a person using a screen reader, which will read the test aloud to a
person using it.

Images with text overlaid are not accessible, so think about your screenshots and make sure they
have accompanying text.

Continuous Improvement

review the documentation, identify missing documentation, or improve documentation that is

frequently used.

This relates to the customer feedback loop. Quickly act on comments from your customers that
tell you your documentation is failing to solve a problem.

Make the time to talk to your support agents about what documentation they might find useful,
and even empower them to create it themselves!

7. Content Discoverability
Consider how customers arrive at your knowledge base in the first place. Very few customers will
consider your knowledge base as a whole, and hardly anyone will arrive at your carefully
constructed homepage.

Consider Every Page is Page One principles as described by Mark Baker. According to EPPO,

fashion as you would with a book.

page for the Web. Wherever you dip your toe into the Web,
Your software documentation is no good if nobody can find it, but there are a number of ways to

The best knowledge base software should be indexable by search engines, with all the correct meta
tags. You should also link to your documentation from your software app, since this is where users
will naturally get stuck.

If possible, make use of contextual help which is served up whenever customers need it.

For example, if customers are having trouble with their billing, ensure a link takes them to a page
with billing documentation that can help solve their problem.

Using Software Documentation Tools

Software documentation tools are specialized tools that are designed to help developers, technical
writers, and other stakeholders create, organize, and manage software documentation. There are
several reasons you should consider making use of specialized tools when creating your software
documentation including:

Automation: Software documentation tools can help automate some of the repetitive and
time-consuming tasks that are involved in creating software documentation. For example,
many software documentation tools can automatically generate documentation from source
code, or from other types of structured data.
Collaboration: Software documentation tools can provide tools and features that make it
easier for teams to collaborate on software documentation. For example, many software
documentation tools provide version control, review and approval processes, and other
features that can help teams work together effectively.
Accessibility: Software documentation tools can make it easier for developers, users, and
other stakeholders to access and use the documentation. For example, many software
documentation tools provide online documentation portals, search tools, and other features
that can help users find the information they need quickly and easily.
There are several different types of software documentation tools that you can use when creating
software documentation. Some examples of types of software documentation tools include:

Source code documentation tools: These tools are designed to help developers
automatically generate documentation from source code. They can parse the source code,
extract comments and other documentation, and generate organized, structured
documentation in a variety of formats.
Collaborative documentation tools: These tools are designed to help teams collaborate
on software documentation. They can provide features such as version control, review and
approval processes, and online collaboration tools that can help teams work together
effectively.
Knowledge management portals: These tools are designed to help users access and use
the documentation. Additionally, they can provide search tools and other features that can
help users find the information they need quickly and easily.
 Knowledge management tools: These tools are designed to help organizations manage
their knowledge assets, including software documentation. They can provide features such
as document management, search and retrieval, and content management that can help
organizations organize, manage, and access their software documentation.
Technical writing tools: These tools are designed to help technical writers create and
manage software documentation. They can provide features such as document templates,
writing aids, and content management that can help technical writers create high-quality,
organized, and consistent documentation.
Source code repositories: These tools are designed to help developers and technical
writers manage their source code and other development artifacts, and they can provide
many features such as version control and collaboration features that are useful for
managing software documentation.

How a Knowledge Base Can Help With Your Software Documentation

A knowledge base can be useful for your software documentation because it can provide a
centralized, organized, and accessible source of information about your software. For example, a
knowledge base can provide a consistent and user-friendly way for end users to access your
documentation, and it can help users quickly find the information they need.

Additionally, a knowledge base can be a useful tool for organizing and maintaining your
documentation. For example, a knowledge base can provide a structure and a process for creating,
reviewing, and updating your documentation, and it can help ensure that your documentation is
accurate, up-to-date, and consistent.

Some features that knowledge base software typically comes with to help with creating effective
software documentation that gets read includes:

Search features. A knowledge base typically provides search and retrieval tools that can
help users find the information they need quickly and easily.
Version control. A knowledge base can help ensure that the documentation is always
accurate and up-to-date by providing tools that can manage different versions of the
documentation.
Analytics. Knowledge base software can come with analytics that can track user behavior,
such as the number of users who access the documentation, the types of information that
users are looking for, and the success rate of finding what they're looking for. This can help
organizations understand how the documentation is being used, and it can help them
improve the documentation to make it more useful for users.

Reference:
1. https://helpjuice.com/blog/software-documentation
3.1.Quality System and Engineering Data Records

Quality System

Software quality product is defined in term of its fitness of purpose. That is, a quality product does
precisely what the users want it to do. For software products, the fitness of use is generally
explained in terms of satisfaction of the requirements laid down in the SRS document. Although
"fitness of purpose" is a satisfactory interpretation of quality for many devices such as a car, a table
fan, a grinding machine, etc.for software products, "fitness of purpose" is not a wholly satisfactory
definition of quality.

Example: Consider a functionally correct software product. That is, it performs all tasks as
specified in the SRS document. But, has an almost unusable user interface. Even though it may be
functionally right, we cannot consider it to be a quality product.

The modern view of a quality associated with a software product several quality methods
such as the following:

1. Portability: A software device is said to be portable, if it can be freely made to work in

various operating system environments, in multiple machines, with other software
products, etc.

2. Usability: A software product has better usability if various categories of users can easily
invoke the functions of the product.

3. Reusability: A software product has excellent reusability if different modules of the

product can quickly be reused to develop new products.

4. Correctness: A software product is correct if various requirements as specified in the SRS

document have been correctly implemented.

5. Maintainability: A software product is maintainable if bugs can be easily corrected as and

when they show up, new tasks can be easily added to the product, and the functionalities
of the product can be easily modified, etc.

Software Quality Management System

A quality management system is the principal methods used by organizations to provide that the
products they develop have the desired quality.

A quality system subsists of the following:

Managerial Structure and Individual Responsibilities: A quality system is the responsibility of

the organization as a whole. However, every organization has a sever quality department to
perform various quality system activities. The quality system of an arrangement should have the
support of the top management. Without help for the quality system at a high level in a company,
some members of staff will take the quality system seriously.

Quality System Activities: The quality system activities encompass the following:
 1. Auditing of projects

2. Review of the quality system

3. Development of standards, methods, and guidelines, etc.

4. Production of documents for the top management summarizing the effectiveness of the
quality system in the organization.

Evolution of Quality Management System

Quality systems have increasingly evolved over the last five decades. Before World War II, the
usual function to produce quality products was to inspect the finished products to remove defective
devices. Since that time, quality systems of organizations have undergone through four steps of
evolution, as shown in the fig. The first product inspection task gave method to quality control
(QC).

Quality control target not only on detecting the defective devices and removes them but also on
determining the causes behind the defects. Thus, quality control aims at correcting the reasons for
bugs and not just rejecting the products. The next breakthrough in quality methods was the
development of quality assurance methods.

The primary premise of modern quality assurance is that if an organization's processes are proper
and are followed rigorously, then the products are obligated to be of good quality. The new quality
functions include guidance for recognizing, defining, analyzing, and improving the production
process.

Total quality management (TQM) advocates that the procedure followed by an organization must
be continuously improved through process measurements. TQM goes stages further than quality
assurance and aims at frequently process improvement. TQM goes beyond documenting steps to
optimizing them through a redesign. A term linked to TQM is Business Process Reengineering
(BPR).

BPR aims at reengineering the method business is carried out in an organization. From the above
conversation, it can be stated that over the years, the quality paradigm has changed from product
assurance to process assurance, as shown in fig.

Engineering Data Records

What Is Data Engineering?

Data engineering is the process of designing and building systems that let people collect and
analyze raw data from multiple sources and formats. These systems empower people to find
practical applications of the data, which businesses can use to thrive.

Data engineering is the complex task of making raw data usable to data scientists and groups within
an organization. Data engineering encompasses numerous specialties of data science.

In addition to making data accessible, data engineers create raw data analyses to provide predictive
models and show trends for the short- and long-term. Without data engineering, it would be
impossible to make sense of the huge amounts of data that are available to businesses.

Why Is Data Engineering Important?

Companies of all sizes have huge amounts of disparate data to comb through to answer critical
business questions. Data engineering is designed to support the process, making it possible for
consumers of data, such as analysts, data scientists and executives, to reliably, quickly and securely
inspect all of the data available.
Data analysis is challenging because the data is managed by different technologies and stored in
various structures. Yet, the tools used for analysis assume the data is managed by the same
technology and stored in the same structure. This rift can cause headaches for anybody trying to
answer questions about business performance.

For example, consider all of the data a brand collects about its customers:

One system contains information about billing and shipping

Another system maintains order history
And other systems store customer support, behavioral information and third-party data
Together, this data provides a comprehensive view of the customer. However, these
different datasets are independent, which makes answering certain questions like what
types of orders result in the highest customer support costs very difficult.

Data engineering unifies these data sets and lets you find answers to your questions quickly and
efficiently.

What Do Data Engineers Do?

Data engineering is a skill that is in increasing demand. Data engineers are the people who design
the system that unifies data and can help you navigate it. Data engineers perform many different
tasks including:

Acquisition: Finding all the different data sets around the business
Cleansing: Finding and cleaning any errors in the data
Conversion: Giving all the data a common format
Disambiguation: Interpreting data that could be interpreted in multiple ways
Deduplication: Removing duplicate copies of data
Once this is done, data may be stored in a central repository such as a data lake or data lakehouse.
Data engineers may also copy and move subsets of data into a data warehouse.

Why Does Data Need Processing through Data Engineering?

Data engineers play a crucial role in designing, operating, and supporting the increasingly complex
environments that power modern data analytics. Historically, data engineers have carefully crafted
data warehouse schemas, with table structures and indexes designed to process queries quickly to
ensure adequate performance. With the rise of data lakes, data engineers have more data to manage
and deliver to downstream data consumers for analytics. Data that is stored in data lakes may be
unstructured and unformatted it needs attention from data engineers before the business can
derive value from it.

important to find software that will automate some of these processes.

The right software stack will extract a huge amount of information and value from your data, which
creates end-to-
through the pipeline, it may be transformed, enriched and summarized several times.
Data Engineering Tools and Skills
Data engineers use many different tools to work with data. They use a specialized skill set to create
end-to-end data pipelines that move data from source systems to target destinations.

Data engineers work with a variety of tools and technologies, including:

1. ETL Tools: ETL (extract, transform, load) tools move data between systems. They access

analysis.
2. SQL: Structured Query Language (SQL) is the standard language for querying relational
databases.
3. Python: Python is a general programming language. Data engineers may choose to use
Python for ETL tasks.
4. Cloud Data Storage: Including Amazon S3, Azure Data Lake Storage (ADLS), Google
Cloud Storage, etc.
5. Query Engines: Engines run queries against data to return answers. Data engineers may
work with engines like Dremio Sonar, Spark, Flink, and others.

Data Engineering vs. Data Science

Data engineering and data science are two complementary skills. Data engineers help make data
reliable and consistent for analysis. Data scientists need reliable data for machine learning, data
exploration, and other analytical projects involving large data sets. Data scientists may rely on data
engineers to find and prepare data for their analysis

The Data Pipeline

There are four key phases of the data pipeline that data engineering directly deals with:

1. Ingestion - This is the task of gathering data. Depending on the number of data sources, ,
this task can be focused or large-scale.
2. Processing - During this phase, ingested data is sorted to achieve a specific set of data to
analyze. For large data sets, this is commonly done using a distributed computing platform
for scalability.
3. Storing - This takes the results of the processing and saves the data for fast and easy
retrieval. The effectiveness of this phase relies on a sound database management system -
which can be on premise or in the cloud
4. Access - Once in place, the data is available to users with access.

Why is Data Engineering important?

If your company lacks a fundamental data engineering strategy, the data that is collected is
essentially useless. Data engineering is a vital aspect of company growth, network interactions,
and predicting future trends.

How Precisely helps data engineers

Tackling the challenge of designing a machine learning model and putting it into production is the
key to getting value back from your big data. But it's also typically the roadblock that stops many
promising machine learning projects.
After the data scientists have done their part, engineering robust production data pipelines has its
own set of challenges. Precisely software helps the data engineer every step of the way.

Precisely's data integration products can help build real-time streaming data pipelines from
multiple sources across the enterprise, including legacy systems such as mainframes. Use
Precisely's data quality solutions to provide entity resolution at scale, to cleanse your big data --
for insights you can trust.

For more information, check out our series of 15-minute webcasts on engineering machine
learning data pipelines:

Pulling in Data from Multiple Sources

Big Data Quality - Data Cleansing at Scale
Tracking Data Lineage from the Source
Data Matching and Deduplication at Scale
Streaming New Data as It Changes

3.2. System Design and Development Data

System design is the phase that bridges the gap between problem domain and the existing system

It is the phase where the SRS document is converted into a format that can be implemented and
decides how the system will operate.

In this phase, the complex activity of system development is divided into several smaller sub-
activities, which coordinate with each other to achieve the main objective of system development.
Inputs to System Design

Statement of work
Requirement determination plan
Current situation analysis
Proposed system requirements including a conceptual data model, modified DFDs, and
Metadata (data about data).

Outputs for System Design

Infrastructure and organizational changes for the proposed system.

A data schema, often a relational schema.
Metadata to define the tables/files and columns/data-items.
A function hierarchy diagram or web page map that graphically describes the program
structure.
Actual or pseudocode for each module in the program.

A prototype for the proposed system.

Types of System Design

Logical Design

Logical design pertains to an abstract representation of the data flow, inputs, and outputs of the
system. It describes the inputs (sources), outputs (destinations), databases (data stores), procedures
(data flows) all in a format that meets the user requirements.

While preparing the logical design of a system, the system analyst specifies the user needs at level
of detail that virtually determines the information flow into and out of the system and the required
data sources. Data flow diagram, E-R diagram modeling are used.

Physical Design

Physical design relates to the actual input and output processes of the system. It focuses on how
data is entered into a system, verified, processed, and displayed as output.

It produces the working system by defining the design specification that specifies exactly what the
candidate system does. It is concerned with user interface design, process design, and data design.

Specifying the input/output media, designing the database, and specifying backup
procedures.
Planning system implementation.
Devising a test and implementation plan, and specifying any new hardware and software.
Updating costs, benefits, conversion dates, and system constraints.

Architectural Design
It is also known as high level design that focuses on the design of system architecture. It describes
the structure and behavior of the system. It defines the structure and relationship between various
modules of system development process.

Detailed Design
It follows Architectural design and focuses on development of each module.

Conceptual Data Modeling

It is representation of organizational data which includes all the major entities and relationship.
System analysts develop a conceptual data model for the current system that supports the scope
and requirement for the proposed system.

The main aim of conceptual data modeling is to capture as much meaning of data as possible. Most
organization today use conceptual data modeling using E-R model which uses special notation to
represent as much meaning about data as possible.

Entity Relationship Model

It is a technique used in database design that helps describe the relationship between various
entities of an organization.

Terms used in E-R model

It specifies distinct real world items in an application. For example: vendor, item,
student, course, teachers, etc.

They are the meaningful dependencies between entities. For example,

vendor supplies items, teacher teaches courses, then supplies and course are relationship.

It specifies the properties of relationships. For example, vendor code, student

name

3.3. Data Accession List (DAL) and Data Criteria List (DCL)

Data Accession List (DAL)

The purpose of the Data Accession List (DAL) is to provide a medium for identifying contractor
internal data that has been generated by the contractor in compliance with the work effort described
in the Statement of Work (SOW). The DAL is an index of the generated data that is made available
on request.

This data item is not a substitute for standard data requirements that are contractually applied.

The DAL is a list of contractor internal data generated by the contractor which the government
MAY request (available upon request) if the need arises. The DAL is not the report, but a list of
their internal data pertaining to this contract which is available upon request
Data Criteria List (DCL)

A data criteria list is a set of standards or guidelines used to evaluate the quality of data.
Here are some common criteria that can be included in a data criteria list:

1. Accuracy: Data should be accurate and free from errors or mistakes.

2. Completeness: Data should be complete and include all relevant information.
3. Consistency: Data should be consistent and not contradict other data in the dataset.
4. Timeliness: Data should be current and up-to-date.
5. Relevance: Data should be relevant to the purpose for which it is being collected
and analyzed.
6. Validity: Data should be valid and measure what it is intended to measure.
7. Reliability: Data should be reliable and produce consistent results over time.
8. Precision: Data should be precise and provide accurate measurements.
9. Accessibility: Data should be easily accessible to those who need it.
10. Security: Data should be secure and protected from unauthorized access or use.

3.4. SE and Development Documentation Sequencing

SE (Software Engineering) and development documentation sequencing refers to the process of

organizing and prioritizing the various documents and artifacts produced during the software
development process. Here are some typical steps in this process:

1. Identify the types of documentation required: The first step is to identify the different types
of documents that need to be produced, such as requirements documents, design documents,
test plans, user manuals, and so on.

2. Determine the sequencing of documentation: Once you have identified the different types
of documents, you need to determine the order in which they should be produced. For example,
requirements documents are typically produced first, followed by design documents, test plans,
and user manuals.

3. Determine the level of detail: Each document should be produced at a level of detail
appropriate for its purpose and audience. For example, requirements documents should be
detailed enough to provide a clear understanding of what the software is supposed to do, while
design documents should be detailed enough to enable developers to implement the software.

4. Define the review and approval process: Each document should be reviewed and approved
by the appropriate stakeholders before it is finalized. The review and approval process should
be clearly defined and documented.

5. Establish version control: Version control is essential for managing the different versions of
each document and ensuring that the latest version is always available to the development team.

6. Create a documentation plan: Finally, you should create a documentation plan that outlines
the sequencing, level of detail, review and approval process, and version control for each
document. The documentation plan should be reviewed and updated regularly to ensure that it
remains relevant and useful throughout the software development process.
3.2. Documentation Levels of Formality

Most people despise preparing documentation. The general viewpoint is documentation is non
value-added bureaucracy. Engineers, in particular, rationalize that if they had wanted to specialize
in documentation, they would have pursued it as a course of study in college. This viewpoint is
contradictory to an engineering development environment that is so dependent on documented data
maturity for decision making.

SE development documentation involves two key decisions:

1. WHAT must be documented?
2. To WHAT DEGREE do you document the details?
The previous discussions addressed WHAT you should
further.

Level of Details

If you ask engineers to prepare a document, they lament about having an impossible task, regardless
of timeframe to complete. The knowledge and maturity of a professional is reflected in the
ntify the key points, and
articulate the results in summary form. In general, if a manager allows you one hour, both parties
must recognize that at the end the hour, you get the one-hour version of the report. Eight hours gets
the manager the eight-hour version an elaboration of details to key points expressed in the one-
hour version.

When you document plans, specifications, and reports, there are key points relevant to the subject
that must be reflected in the document outline. These points, in turn, require various levels
of details. The key points are:

1. Engineers need to learn to identify WHAT information (i.e., major points) is required to be
communicated.
2. Apply common sense to scale the level of detail to fit the available time and resources.

Key Point If you have addressed the critical points, anything else should be just supporting detail.
The ability to scale these levels of detail and still address the key points depends on personal
knowledge, experience and seasoned maturity

3.3. Export Control of Sensitive Data and Technology

Today the Internet provides a mechanism for immediate data communications and access between
organizations in-country as well as internationally. As a result, the Internet offers tremendous
opportunities for contract programs to exploit the technology by establishing Web sites that enable
authorized organizations and individuals who are geographically dispersed to post and access
technical program information. This environment, when uncontrolled, however, adds new
dimensions and threats to information access.
People often confuse EXPORT CONTROL information with security classification systems.
Although both are certainly interrelated, EXPORT CONTROL information may be sensitive but
unclassified. Classified data handling and procedures are yet another issue.

Export Control of Technology

Many people erroneously believe that a technology or information export to a foreign national,
organization, or country occurs when that information is physically transferred outside the country.

The fact is technology transfer also occurs in-country with foreign nationals, whether direct or via
the Internet. Technology transfer is governed in the United States by EXPORT CONTROL laws
and regulations such as the International Traffic and Arms Regulations (ITAR). So, what does this
mean to SE?

If you:
1. Post EXPORT CONTROL technology or information to an uncontrolled Web site.
2. E-mail the information via the Internet.
3. Simply give that information to foreign nationals, organizations, or governments without being
licensed and taking reasonable measures to restrict access to only authorized users,
you have violated the US ITARS governance.

Warning! ALWAYS consult EXPORT CONTROL Officer and security,

legal, and contracts organizations before initiating an action that may violate EXPORT CONTROL
or security regulations and procedures.

The US government and other countries have explicit laws and regulations that govern the
EXPORT of technology and data to foreign nationals, organizations, and countries. The laws carry
severe penalties for noncompliance.

3.4. System Documentation Issues

System documentation has a number of issues that SEs must address. Here are several issues that
commonly challenge many programs:

Issue 1: Data validation and authentication

Issue 2: Posting acquirer and vendor documentation
Issue 3: Proprietary data and nondisclosure agreements
Issue 4: Vendor-owned data
Issue 5: Electronic signatures

Issue 1: Data Validation and Authentication

Since SE technical decision making is predicated on data integrity, the challenge is determining if
external and internal data are valid and authentic. This includes DCL items for models and
simulations, interfaces, and test data. Additionally, production contracts may require using SE
documentation created by the original vendor. So, how do you determine the validity and
authenticity of the data?
Data validity and authenticity require rigorous investigation, by inspection, sampled testing,
Functional Configuration Audit (FCA), or Physical Configuration Audit (PCA). Data integrity is
particularly troublesome when the original System Developer delivered the Product Baseline and
transferre

Issue 2: Posting Acquirer and Vendor Documentation

Many engineers erroneously believe that they can arbitrarily copy and post Acquirer and vendor
documentation on a program Web site. Although Acquirer solicitation and other data may be posted
for public access, DO NOT copy this material and POST it unless your organization has prior
written authorization from the Acquirer or Vendor. Posting creates configuration management
control, proprietary, data concurrency, and copyright issues. If you need to provide program
personnel access to this data, simply provide links from the program Web site to the Acquirer or
vendor Web site, assuming they approve. This avoids ownership, proprietary, concurrency,
copyright, and other issues.

Issue 3: Proprietary Data and Non-Disclosure Agreements

Finally, before any EXCHANGES of data can occur between any organization and the Acquirer,
User, subcontractors, and vendors, execute Proprietary Data Agreements and/or Nondisclosure
Agreements between the parties.

Referral ALWAYS consult with the Program Director and/or Technical Director as well as the
contracts, legal, and EXPORT CONTROL organizations for the proper procedures for:

1. Communicating and exchanging data with external organizations.

2. Establishing proprietary data agreements and/or nondisclosure agreements.

Issue 4: Vendor-Owned Data

When Acquirers procure a system, they are often willing to trade off documentation funds for
system capabilities, especially when they have limited budgets. Later, if they decide to procure
system upgrades that are dependent on these data, they shift the burden and risk of obtaining the
data to the System Developer. If the original developer and new System Developer are competitors,
this creates a very challenging problem for programs and SEs have to solve it.

The challenge is one of cost and schedule. Programs inevitably underestimate the amount of
resources required to acquire documentation owned by other organizations. Thoroughly investigate
these issues during the proposal phase and establish data exchange agreements, terms, and
conditions prior to Contract Award. As an Acquirer, if you wait until after Contract Award, guess
WHO is in the POWER position to control the negotiation? The supplier is and you can rest assured
the procrastination will cost you significantly!

Issue 5: Electronic Signatures

When implementing an integrated data environment IDE, one of the key issues is establishing a
secure method for SE documentation reviewers to electronically approve the documents. This
s, and tools be established to ensure that only
authorized approvers can review and release documentation for implementation.