Unit 6: Data and Analysis — Long Questions

Data science is a field that focuses on collecting, analyzing, and using data to find useful information, converting raw data into meaningful knowledge. It involves collecting data from different sources, cleaning and organising it, analysing it, and carefully interpreting the results so that questions can be answered using facts rather than guesswork.

Data science is interdisciplinary because it combines several fields: mathematics (for numbers and calculations), statistics (for analysing and summarising data), computer science (for tools and programming skills), domain knowledge (for understanding real problems), and communication skills (for explaining results clearly). Each field supports the others, making data analysis more effective.

The overall workflow of data science (Figure 6.1) follows a cycle: Data Collection (gathering and processing) → Data Analysis (exploring and cleaning) → Modeling (training and testing) → Decision Making (insights and actions), with a feedback loop back to data collection so the process keeps improving.

Importance in modern applications data science supports decision-making in many areas of life, helps predict future trends, and reduces guesswork. It is widely used in business (customer insights, fraud detection, sales forecasting), education, and healthcare (medical diagnosis, predictive analytics), and modern technology depends heavily on it to turn data into knowledge that solves real-world problems.

Data is a collection of raw facts and values that produces information; it can be in the form of numbers, text, images, audio, or video. There are two main types of data:

1. Structured data: organised in a clear format such as tables or spreadsheets, with rows and columns that are easy to understand. Examples include students' mark sheets and databases.
2. Unstructured data: does not follow a fixed format. Examples include emails, images, audio, and videos. It is difficult to analyse directly and needs special tools to process it.

Both types of data are important, and modern systems use both. Understanding the type of data is important before analysis, since poor data understanding can lead to errors, and data must be accurate and reliable to produce correct results.

Data sources and collection methods: Data can be collected from many sources, including surveys, interviews, forms, sensors, and online platforms such as blogs and social websites. Manual data collection involves human effort, whereas automatic data collection uses machines and software. Data can also be collected in real time or stored over time, and the method chosen depends on the purpose of the analysis. Good data sources and accurate data collection improve the quality of analysis.

Once collected, data must also be properly stored and managed — saved in files, databases, or cloud systems, organised for easy use, protected with security and backup systems — since well-managed data improves efficiency and supports long-term use.

Machine learning is used in many areas of modern life to help organisations analyze large amounts of data quickly, reducing human effort and saving time (Figure 6.5).

Healthcare Machine learning helps analyze medical data and supports disease detection and diagnosis — for example, detecting cancerous cells in medical images, identifying early signs of Alzheimer's disease or dementia, and assisting in diagnosing cardiovascular diseases, helping doctors plan timely treatments. It also supports predictive analytics and drug discovery.

Business Businesses use machine learning to understand customer behaviour, predict sales and market trends, and manage inventory. Examples include detecting fraud in financial systems, customer service through chatbots, and suggesting products to customers, making marketing more effective and decisions more data-driven.

Education Machine learning improves learning systems by tracking students' performance to improve their grades, providing personalized learning content, and making attendance and assessment systems smarter, through adaptive learning and tutoring systems.

Daily life Machine learning is used in mobile phone face recognition, voice assistants that understand spoken commands, personalized social media content, spam filtering in email, navigation route suggestions, and recommendations for videos and music, as well as smart home devices — making daily tasks easier and faster.

Building a machine learning model involves several clear, cyclic steps (Figure 6.6: Select Data → Train Model → Test Model → back to Select Data):

1. Select/collect data: Data relevant to the problem is collected — for example, for student grade prediction, data about attendance, quiz marks, assignment scores, study hours, and previous exam results is gathered, then cleaned and prepared for learning.

2. Feature selection: Important values, called features, are selected because they help the model understand the factors that affect the outcome (the target value). For example, when predicting student marks, study hours is the feature and the marks are the target value. Correct feature selection improves learning, while unnecessary features can reduce accuracy.

3. Feature engineering: This means improving data for better learning — selecting, modifying, and creating features, since raw data is often not ready for models and may need cleaning and scaling. For example, raw study-hours data could be cleaned, scaled, and a new feature such as "study hours per day" could be created to improve predictions.

4. Train-test split: The prepared data is divided into a training set, which teaches the model patterns, and a testing set, which checks the model's performance on unseen data. A common split is 80% training and 20% testing; this split prevents overfitting and shows the model's real performance.

5. Train and test the model: The model (e.g., a linear regression model) is trained on the training data and then tested on the test data; its performance is evaluated (e.g., using Mean Squared Error), and based on results and new data, the model's accuracy can be further improved.

Once trained and tested, the model can predict outcomes such as a student's likely grade from new input data, helping identify students needing extra support.

Model evaluation is the process of checking how well a machine learning model performs. It measures the quality of predictions made using test data not used during training, which helps identify the model's strengths and weaknesses. Different performance metrics are used:

Accuracy shows how many predictions are correct out of the total predictions, measured as:
Accuracy = (TP + TN) / (TP + TN + FP + FN)
where TP = True Positives (correctly predicted positive), TN = True Negatives (correctly predicted negative), FP = False Positives (incorrectly predicted positive), FN = False Negatives (incorrectly predicted negative). High accuracy means better performance, but accuracy alone can be misleading, especially with unbalanced data.

Precision measures how many predicted positive results are actually correct (focuses on correctness):
Precision = TP / (TP + FP)

Recall measures how many actual positive cases are correctly identified (focuses on completeness):
Recall = TP / (TP + FN)

F1-score the balance between precision and recall, giving a single performance score, especially useful when data is unbalanced:
F1-Score = 2 × (Precision × Recall) / (Precision + Recall)

Worked example Given TP = 40, TN = 50, FP = 10, FN = 5:
Precision = 40 / (40+10) = 40/50 = 0.8
Recall = 40 / (40+5) = 40/45 = 0.8889
F1-Score = 2 × (0.8 × 0.8889) / (0.8 + 0.8889) = 2 × 0.7111 / 1.6889 = 0.8421

Thus Precision = 0.8, Recall = 0.8889, F1-Score = 0.8421. Since high accuracy may still hide wrong results in some cases, precision, recall, and F1-score are used together for a fuller, fairer evaluation of model performance.

Validation is used to check how well a model works on new data. It is carried out after training and testing to confirm the model's reliability, and it prevents the model from merely memorizing (overfitting to) the training data, so it supports fair evaluation.

Validation dataset preparation A validation dataset is kept separate from the training and testing data and is used only to check model performance; the model does not learn from this data. Validation data must be clean, accurate, and balanced to represent real conditions, and it helps detect overfitting and increases confidence in the model.

Model robustness and reliability Robustness means the model performs well in different situations and handles noise in data without failing from small changes. A reliable model gives consistent results, and testing under different conditions checks this robustness, which is important for real-world use and builds trust in the system.

Model improvement using hyperparameter tuning: Hyperparameter tuning means adjusting the model's settings (parameters) that control its learning behaviour, to improve accuracy and reduce errors. Different parameter values are tested, with validation data guiding the tuning process. This is similar to adjusting the volume of a speaker: if hyperparameters are set too high or too low, the model performs poorly (just like sound becomes distorted if too loud or inaudible if too soft), but when set correctly, the model learns better and produces more accurate results. Proper tuning improves performance, while poor tuning reduces effectiveness.