Print Friendly, PDF & Email

Data processing provides a connection between data capture and data output. In this article, we look at 10 key things everyone needs to know about scientific data processing.

1. Data Needs Cleaning

If not cleaned properly, errors in data can contribute to severely inaccurate results. All data sets will have errors. The bigger the data sets, the more possible errors creep in.

Errors can come from incorrect experiments, incorrect capture, or incorrect entry. User entries can contain complexities like abbreviations, misspellings, and varied terms for non-varied entities.

Data containing errors can’t be used effectively. One of the key jobs of the data processing scientist, that must be done right, is to clean data, that is, to effectively and efficiently remove errors from data so that it is useful. It’s not as exciting or glamorous as analytical models, but it is vitally important for the latter to work, and must be done right.

2. No Model is Perfect

When you’ve invested a lot of time developing a model from strict analytical procedures, a vested interest in the process tends to provide us with an overly positive view of the results.

It’s important to remember that all models are just assimilation, approximations, and estimations. They might be only an estimated 1% chance for error, but this in itself is only an estimation. It is very important to remember that modeling is only a predictive tool, theoretical, and looking at possible outcomes, not a scientific absolute.

3. Approximate Can Sometimes Be Good Enough

A recent study detailed how a major entertainment company sped up its data processing time in huge proportions by using an APPROX_DISTINCT(x) function in place of COUNT(DISTINCT x).

The application is amazingly as simple as it looks, even to non-programmers, one is approximate, one is exact. With data sets in the millions, this replacement resulted in only a 4% error on data, which for the purpose was quite acceptable, at the same time staggeringly reducing computing time from hours to minutes.

Reduction in computing time of this level can save the company huge amounts on hardware, and increase profits by bringing valuable data to where it’s needed far quicker, with only a small error rate.

As data scientists, we can sometimes get bogged down by trying to ensure accuracy to within 99.9%, when even in straight descriptive data applications, approximations are often good enough.

3. Big Data is a Tool, Not a Solution

So much focus is put on “Big Data” that we sometimes forget it is merely a material resource to work with, not a solution. Big Data needs processing like any other data. Even more so.

Although Big Data provides lots of insights, once transformed, it still needs cleaning and transforming. It’s vitally important to remember the place of Big Data in data science.

4. Transforming Data Should Not Rely on One Technique

While it’s tempting to chase the latest shiny object, data science, as the name implies, relies on a scientific approach. When any technique is favored above a comprehensive approach, then the scientific method breaks down.

When scientific data processing is applied, the analyst uses a variety of approaches to draw conclusions. The more comprehensive the approach, the more sound the results will be.

5. Break Down Data Into Subsets

Only when data is broken down can relationships start to be seen. An important facet of scientific data processing is the concept of breaking down data. Once the subsets of data are developed, new trends and hypotheses can be developed.

6. Relationships are Important

When processing data, recognizing data relationships is an imperative skill. Relationships are the key to developing data models and discovering truths behind the data that will benefit management and users.

Without finding a relationship, the data is meaningless. Although relational databases have categories that relate, to draw new discoveries, there needs to be a method of finding new relationships between data sets.

7. Non-relational Databases are Also Important

Non-relational databases are more difficult to query, but have the power to section, model, and analyze both structured and unstructured data. With large relational databases, there is potential for larger errors in fields such as object-relational impedance mismatch. On the other hand, non-relational databases maintain more pure sets of data where objects can overlap easily.

Non-relational databases are an important concept in scientific data processing, especially in the growing sizes of Big Data.

8. Correlation and Causation Are Not the Same Thing

While establishing relationships, categorizing variables, and breaking down data into variables are important, all of these areas can introduce an error in matching correlation with causation.

The same result doesn’t indicate the same cause. When we draw parallels from two sets of data to establish a pattern, we may include variables that do not in fact match. This is where an analyst’s skill in scientific data processing is required.

Two sets of data with the same trends don’t imply the same cause. A simple, yet humorous example is the correlation that most people who floss are not obese, so flossing causes weight loss.
To avoid spurious correlation errors, data processing scientists need to ensure the data is considered from as many angles as possible, and most importantly, it’s the human input here that can often see something a machine cannot.

Human logic recognizes that flossing doesn’t make you thin, data processor’s intuitive knowledge needs to be used to view data sets and add quantifiers where needed in the presentation of facts.

9. To Be Scientific, Data Processing Needs Bayesian Logic

Data sets and statistics can be used, or more correctly, misused to prove anything. A Bayesian approach will help ensure accuracy.

What Does Bayesian Logic Mean?

Bayesian logic is based on Bayes’s model of error and means disproving as well as proving. If we prove a theory, this is fine, but it is only a theory. We need to consider the chance of error based on all possible outcomes, which means we need to consider the chance of alternative theories.

A simple explanation of the power of Bayesian logic is to consider conducting an experiment which involves blindly drawing different colored balls from a basket. Although improbable, it’s entirely possible that the same color is drawn every time. Without considering the Bayes model, we may wrongly conclude there are no other colors.

10. A Good Data Scientist Asks Lots of Questions

The last, and most important, point to remember about scientific data-processing is that questions need to be asked. Don’t assume!

A mental model, which is developed by good question and answer tactics, is the starting point of all good data analysis. If one doesn’t ask the right questions, one will not receive the right answers.

It will be pointless spending large amounts of time on a data set if one starts with the wrong assumptions because they did not obtain all the facts. Only with good quality questions can one find good quality answers.

A good data scientist is one who asks questions.


Everyone has the ability to benefit from scientific data processing. We need only remember that scientific data processing is a science when it adheres strictly to scientific principles. It’s important that work is quantitative, qualitative, and fact-based. Keep asking questions, and keep validating, and use all the resources in your toolbox.

Previous articleWhat is the Role of Big Data in Data Analytics?
Next articleHow Data Analytics Prevents Delayed Deliveries?
I am a professional author and blogger. I initially trained as a mechanical engineer, majoring in mathematics, then spent most of my life in the clouds as a professional pilot. Throughout my working life, I have always loved writing, my passion for literacy in education led to the development of a free children's literature website. I particularly enjoy writing on technical topics, my writing achievements include producing a number of popular textbooks for pilots. My hobbies are playing the cello, golf, and writing children's books.


Please enter your comment!
Please enter your name here