What big data in manufacturing means
Big data in manufacturing is the use of large and varied data from machines, quality, maintenance and logistics to find patterns and support decisions. The value is not the size of the database but the question it answers: why the line stopped, where scrap starts, when a machine will fail.
The idea is usually described by the "Vs": volume, velocity, variety, veracity (data quality) and value. On the shop floor, variety and veracity matter more than volume.
Visual map of manufacturing big data
The map shows the chain collect, store, analyze and act, with the numbers from this article's example.
From data to decision: 4 steps
- Collect: sensors, PLCs, MES, ERP, quality spreadsheets and manual logs. Connecting machines is covered in industrial IoT (IIoT).
- Store: a historian for time series, a relational database for transactions and, where variety is high, a data lake for raw data.
- Analyze: from descriptive ("what happened") to diagnostic ("why"), predictive ("what will happen") and prescriptive ("what to do").
- Act: an alert, a maintenance order, a parameter change or a new standard work. Without action, analytics is just a report.
Example 1: how much data a line generates
Assume 50 sensors read once per second, at 16 bytes per reading (value, timestamp and tag ID, an assumption of the example).
| Step | Math | Result |
|---|---|---|
| Readings per day | 50 sensors × 86,400 s | 4,320,000 |
| Size per day | 4,320,000 × 16 bytes | 69.12 MB |
| Size per year | 69.12 MB × 365 | 25.2 GB |
| 20 machines, 1 year | 25.2 GB × 20 | 504.6 GB |
| At 1 reading per minute | 4,320,000 ÷ 60 | 72,000 readings/day |
The result is surprising: 25.2 GB per year per machine fits on an ordinary server. For 20 machines, about 504.6 GB per year is still manageable. Logging slow signals once per minute cuts volume by 60 times without losing what matters.
Daily volume = sensors × readings per second × 86,400 × bytes per reading = 50 × 1 × 86,400 × 16 = 69.12 MB
The lesson: before buying a "big data" platform, calculate your own volume. Many plants need clean, well-labeled data, not petabytes.
Example 2: from downtime data to a decision
Fictional example: automatic downtime logging in one cell added up to 900 minutes in a month.
| Downtime cause | Minutes | % | Cumulative % |
|---|---|---|---|
| Material shortage | 310 | 34.4 | 34.4 |
| Tool changeover | 240 | 26.7 | 61.1 |
| Process adjustment | 150 | 16.7 | 77.8 |
| Breakdown | 120 | 13.3 | 91.1 |
| Other | 80 | 8.9 | 100.0 |
| Total | 900 | 100.0 |
Check: 310 + 240 + 150 + 120 + 80 = 900. Shares: 310 ÷ 900 = 34.4%; 240 ÷ 900 = 26.7%; 150 ÷ 900 = 16.7%; 120 ÷ 900 = 13.3%; 80 ÷ 900 = 8.9%. The top two causes add up to 550 minutes, or 61.1% of downtime.
The decision comes from the data: attack material shortage (supply, kanban) and tool changeover first. That is the logic of a Pareto chart; big data only automated the collection. The effect shows up later in OEE.
Data warehouse or data lake?
| Criterion | Data warehouse | Data lake |
|---|---|---|
| Data | Structured, modeled | Raw, any format |
| Typical use | Reports and fixed KPIs | Exploration, statistical models, long time series |
| Upfront effort | Model before loading | Fast load, organize later |
| Risk | Rigidity | Becoming a "data swamp" without a catalog |
Many plants combine both: raw data in the lake, curated KPIs in the warehouse, and a screen like the production dashboard at the end.
Use cases that tend to pay off
- Predictive maintenance: vibration and temperature to anticipate failures (see AI in predictive maintenance).
- Quality: link process parameters to defects and use real-time SPC.
- Planning: forecast demand and consumption from historical series.
- Energy: find peaks and wasteful hours.
- Simulation: feed a digital twin with real data.
Data quality: the real bottleneck
- Unique identification: the same machine, product and downtime reason codes in every system.
- Synchronized clocks: without correct time you cannot match machine events to batches.
- Standard downtime reasons: "other" cannot be the second largest category.
- Data owner: one person responsible for each source.
To start, pick one problem, such as the downtime above, and build the full path, as in digital transformation in 5 steps.
Common mistakes
- Buying a platform before defining the business question.
- Collecting everything and cleaning nothing.
- Storing 1-second data where 1-minute data would do.
- Delivering a report with no owner for the action.
- Ignoring manual logs and the discipline of filling them in.
- Confusing correlation with cause: validate with a test and with the shop floor.
FAQ
What is big data in manufacturing?
It is the use of large, varied data from machines, quality, maintenance and logistics to find patterns and support production decisions.
What are the Vs of big data?
Volume, velocity, variety, veracity and value. On the shop floor, variety and veracity (quality) often matter more than volume.
Does my plant need big data?
Only if you have a question current data cannot answer. In the example, 50 sensors per machine generate about 25.2 GB a year, which an ordinary server handles.
What is the difference between big data and business intelligence?
BI shows predefined KPIs, usually from structured data. Big data adds varied sources and exploratory analysis, such as predictive models.
Where do I start?
Pick one measurable problem, such as downtime or scrap, connect the needed sources, clean the data, analyze with Pareto or SPC and define an action with an owner.
How does it relate to Industry 4.0?
Big data turns the data collected by IIoT and MES into decisions. Connectivity without analysis only produces volume.
Sources
- LANEY, D. 3D Data Management: Controlling Data Volume, Velocity and Variety. META Group, 2001.
- KLEPPMANN, M. Designing Data-Intensive Applications. O'Reilly, 2017.
- KIMBALL, R.; ROSS, M. The Data Warehouse Toolkit. Wiley.
- MAYER-SCHÖNBERGER, V.; CUKIER, K. Big Data: A Revolution That Will Transform How We Live, Work, and Think. Houghton Mifflin Harcourt, 2013.
Keep studying: trusted sources and links
- Plattform Industrie 4.0
- SENAI: educação profissional e tecnológica para a indústria (in Portuguese)
- Wikipedia: Fourth Industrial Revolution
Save this address: Manufatura Enxuta is your school for lean manufacturing. Come back whenever you need to study, with a complete article, a visual map and trusted sources. Follow one of the learning paths or check the A–Z glossary.
Want to measure what really matters?
Download the free e-book on performance indicators and track the losses in your operation.
Download the e-book
About the author
Lean Manufacturing & Behavioral Management Specialist
Over 20 years in the automotive and metalworking industries (GM and Dana), Lean Manufacturing practitioner since 2006. SENAI instructor and mentor in Brazil’s Brasil Mais Produtivo program, delivering consulting, training and audits for 50+ companies, combining quality, productivity and people development.
