Getting mean and standard deviation from a histogram

Calculating Mean and Standard Deviation from Histogram Data in MATLAB/Octave

Learn how to accurately derive the mean and standard deviation from histogram bin counts and edges, a common task in data analysis when raw data is unavailable.

Histograms are powerful visual tools for understanding the distribution of a dataset. Often, you might be presented with histogram data—bin counts and bin edges—rather than the original raw data. In such scenarios, calculating statistical measures like the mean and standard deviation requires a specific approach. This article will guide you through the process of accurately estimating these statistics directly from histogram information using MATLAB or Octave.

Understanding Histogram Data Structure

Before diving into calculations, it's crucial to understand how histogram data is typically structured. A histogram is defined by its bins, which are contiguous intervals, and the count of data points falling into each bin. In MATLAB/Octave, the histcounts function (or hist for older versions) returns two key pieces of information:

1. N (or counts): A vector where each element represents the number of data points in a corresponding bin.
2. edges (or x): A vector defining the boundaries of the bins. If there are k bins, there will be k+1 edges. The i-th bin spans from edges(i) to edges(i+1).

flowchart TD
    A[Raw Data] --> B{Binning Process}
    B --> C[Bin Edges (k+1 elements)]
    B --> D[Bin Counts (k elements)]
    C & D --> E[Histogram Representation]
    E --> F{Calculate Mean & Std Dev}
    F --> G[Estimated Statistics]

Flowchart illustrating the process from raw data to estimated statistics via histogram.

Calculating Bin Centers

The first step in estimating the mean and standard deviation from histogram data is to determine the 'center' or 'midpoint' of each bin. We assume that all data points within a given bin are concentrated at its center for the purpose of these calculations. This is a reasonable approximation, especially for a large number of bins or a relatively smooth distribution.

If your bin edges are [e1, e2, e3, ..., ek, ek+1], then the center of the first bin is (e1 + e2) / 2, the second is (e2 + e3) / 2, and so on. Generally, the center of the i-th bin is (edges(i) + edges(i+1)) / 2.

% Assuming 'counts' is the vector of bin counts and 'edges' is the vector of bin edges

% Calculate bin centers
bin_centers = (edges(1:end-1) + edges(2:end)) / 2;

MATLAB/Octave code to calculate bin centers from bin edges.

Estimating Mean and Standard Deviation

With the bin centers and their corresponding counts, we can now estimate the mean and standard deviation. These calculations are essentially weighted averages, where the bin centers are the values and the bin counts are their respective weights.

Estimated Mean

The estimated mean ((\bar{x})) is calculated as the sum of each bin center multiplied by its count, divided by the total number of data points (which is the sum of all counts):

[ \bar{x} = \frac{\sum_{i=1}^{k} (c_i \cdot n_i)}{\sum_{i=1}^{k} n_i} ]

Where (c_i) is the center of the (i)-th bin and (n_i) is the count in the (i)-th bin.

Estimated Standard Deviation

The estimated standard deviation ((s)) is a bit more involved. It's calculated using a weighted variance formula:

[ s = \sqrt{\frac{\sum_{i=1}^{k} n_i \cdot (c_i - \bar{x})^2}{(\sum_{i=1}^{k} n_i) - 1}} ]

Note the (N-1) in the denominator for the sample standard deviation, where N is the total sum of counts. If you're treating the histogram as a population, you would use N instead of N-1.

% Assuming 'counts' and 'bin_centers' are already defined

% Total number of data points
N_total = sum(counts);

% Calculate estimated mean
estimated_mean = sum(bin_centers .* counts) / N_total;

% Calculate estimated standard deviation
% Using (N_total - 1) for sample standard deviation
estimated_std = sqrt(sum(counts .* (bin_centers - estimated_mean).^2) / (N_total - 1));

fprintf('Estimated Mean: %.4f\n', estimated_mean);
fprintf('Estimated Standard Deviation: %.4f\n', estimated_std);

MATLAB/Octave code for calculating estimated mean and standard deviation.

Complete Example in MATLAB/Octave

Let's put it all together with a practical example. We'll generate some random data, create a histogram, and then derive the statistics from the histogram data.

% 1. Generate some sample data
raw_data = randn(1, 1000) * 5 + 10; % 1000 points, mean 10, std 5

% 2. Create a histogram and get counts and edges
num_bins = 20;
[counts, edges] = histcounts(raw_data, num_bins);

% 3. Calculate bin centers
bin_centers = (edges(1:end-1) + edges(2:end)) / 2;

% 4. Calculate estimated mean and standard deviation
N_total = sum(counts);
estimated_mean = sum(bin_centers .* counts) / N_total;
estimated_std = sqrt(sum(counts .* (bin_centers - estimated_mean).^2) / (N_total - 1));

% 5. Compare with actual statistics from raw data
actual_mean = mean(raw_data);
actual_std = std(raw_data);

fprintf('--- Histogram-derived Statistics ---\n');
fprintf('Estimated Mean: %.4f\n', estimated_mean);
fprintf('Estimated Standard Deviation: %.4f\n', estimated_std);
fprintf('\n--- Actual Raw Data Statistics ---\n');
fprintf('Actual Mean: %.4f\n', actual_mean);
fprintf('Actual Standard Deviation: %.4f\n', actual_std);

% Optional: Visualize the histogram and estimated mean/std
figure;
histogram('BinEdges', edges, 'BinCounts', counts);
hold on;
plot([estimated_mean estimated_mean], ylim, 'r--', 'LineWidth', 2, 'DisplayName', 'Estimated Mean');
plot([estimated_mean - estimated_std, estimated_mean + estimated_std], [max(ylim)/2, max(ylim)/2], 'g-', 'LineWidth', 2, 'DisplayName', 'Estimated Std Dev Range');
legend('show');
title('Histogram with Estimated Mean and Std Dev');
xlabel('Value');
ylabel('Frequency');
hold off;

Full MATLAB/Octave example demonstrating calculation and comparison.