The linear response observed in high-end instrumentation should be a baseline expectation for scientific measurement. However, in the commercial spectroscopy market, vendors often "gloss over" non-linear hardware performance by promising a software-side fix.

FIGURE 1. Linearity Comparison under varying exposure.PART A: TCD1304 with High-Slew AFE and 16-bit Differential ADC. This demonstrates an inherently linear response driven by signal chain integrity. Even with standard timer-driven clocks, the high-bandwidth AFE prevents the dV/dt \"rounding\" of peaks, ensuring the data remains metrologically honest (0.19% INL).PART B: Popular Commercial USB Spectrometer. This displays the characteristic slew-limited \"S-curve.\" This illustrates a hardware bottleneck where the internal signal chain cannot track rapid transitions—a failure of their design that global software correction cannot truly rectify.

1. The Mathematical "Canard"

As seen in the figure above, the "S-curve" response on the right (b) is often presented as a minor calibration hurdle. This is a metrological "canard" that ignores the failure of their design. While vendors claim these errors can be fixed in post-processing, they are actually masking a signal chain that cannot keep up with the physical reality of the light being measured.

There is a widespread practice of claiming that hardware non-linearity can be corrected through the pixel-wise application of a global polynomial:

S(Aₙ) = a₀ + a₁Aₙ + a₂Aₙ² + a₃Aₙ³ + ...

In this model, Aₙ is the raw intensity of pixel n, and the coefficients aᵢ are assumed to be constant across the entire array. While mathematically convenient, this model is fundamentally out of sync with the reality of any instrument where the response is limited by bandwidth or slew.

2. The Smoking Gun: The Mercury Line Test

To demonstrate the failure of this global model, we evaluated the coefficients required to linearize the response of the commercial instrument shown in (b) using two mercury (Hg) lines: the broad 546nm peak and the narrow 435nm peak.

Inconsistency of Polynomial Correction

Spectral Line	a0​ (Constant)	a1​ (x1)	a2​ (x2)	a3​ (x3)
Broad Peak (546nm)	-0.0112	1.1692	-0.6160	0.6320
Narrow Peak (435nm)	-0.0343	1.5597	-2.0373	1.7793

The results are stark: the narrow, sharper peak requires second- and third-order coefficients that are nearly triple those of the broader peak. Furthermore, the 23.1 mV gap in the constant term (a0​) confirms that the "correction" is not a static property of the pixel, but a dynamic function of the signal’s geometry.

3. The Physical "Why": Slew-Limited Response

The divergence in these coefficients proves that the error is tied to the steepness (the dV/dt) of the spectral line. In a slew-limited instrument, non-linearity is a dynamic result of how the signal changes across a set of pixels. Because a global polynomial is static, it is mathematically irreconcilable with the physical behavior of the sensor signal chain.

4. Hardware Integrity Over Software "Gloss"

The goal of our open-science project is to provide metrological and radiometric truth at the hardware level, eliminating the need for mathematical "guesses". By using a hardware-locked state machine architecture and a sub-millivolt noise floor, we have achieved 0.19% Integral Non-Linearity (INL) without software trickery.

Our mission is to provide affordable, high-fidelity hardware to under-resourced scientists who cannot afford to rely on "black box" commercial corrections that fail under scrutiny.

5. The Proof of Radiometric Truth

As demonstrated in Figure 2, the difference between a high-integrity signal chain and a software-dependent one is unmistakable. In our instrument (a), normalizing the spectral traces by exposure time results in a perfect overlay across the dynamic range. This suggests that the system is capturing radiometric truth; the data scales linearly with the photon count because the high-slew AFE and 16-bit differential ADC preserve the peak geometry without "funny business" in the baseline. Conversely, the commercial unit (b) shows significant divergence. This inconsistency confirms that their "S-curve" is not a simple scaling factor but a dynamic distortion that varies with intensity, making it physically impossible to reconcile through standard normalization or global polynomial correction.

FIGURE 2. Intensity Normalization and Baseline Stability.PART A: TCD1304 with High-Slew AFE/ADC. When normalized by exposure time, all spectral traces OVERLAY PERFECTLY across the entire dynamic range. This confirms the signal chain is metrologically honest, maintaining a stable dark floor and linear scaling (0.19% INL) without \"funny business\" in the baseline.PART B: Popular Commercial USB Spectrometer. Even with dark subtraction, the normalized intensities remain HIGHLY INCONSISTENT. The failure of the internal design to track peak geometry leads to a divergent response where the data cannot be reconciled simply by dividing by exposure time.

6. Closing: Open Science in Practice

The goal of this project has been to raise the bar and provide high-fidelity, linear spectroscopy in an open-source design at a fraction of the cost of commercial units. By prioritizing hardware integrity and radiometric truth, we have moved away from metrological fictions and toward an instrument that any scientist can trust.

Full documentation, including the hardware-locked state machine architecture and 0.19% INL performance data, is available on GitHub for the community to pick up, use in their research and advance the standard of open-source instrumentation:https://github.com/drmcnelson/TCD1304-Sensor-Device-with-Linear-Response-and-16-Bit-Differential-ADC

In Part I of this project (detailed in the GitHub README), we characterized the architecture that achieved 0.2% INL and eliminated 'saturation sag' by moving to a hardware-locked timing engine.

If you are interested in the front-end architecture and the characterization that achieved 0.2% INL, you can find the Part I discussion and the original 0.19% validation data here: [https://www.reddit.com/r/Optics/comments/1snfrz6/linear_ccd_instrumentation_achieves_019_inl/]"

Here in Part II, we are going to talk about residual charge transfer (RCT) and how to eliminate "ghosting" - a phenomenon common to imaging detectors. This is another critical part of achieving radiometric truth. Solving the RCT problem helps you to be able to publish data that is reproducible and valid.

Here is what the RCT problem looks like in a system that has good pulse drivers for the shift gate that harvests charge from the detector region of the sensor into the analog shift register for readout. These are spectra of a green LED at different intensities each collected with a 20 msec exposure and immediately followed by a frame with the LED turned off (orange). The intensity appearing in the second frame is due to charge that was not harvested in the first assertion of the shift gate. Notice that the high and low intensity in the initial frame produce similar levels in the RCT frame. This is because the initial readout is helped by a self-field effect and we are using a very good driving circuit so that we are able to get almost all of the "easy" charge in the first gate. But the residual is still significant and can be an issue if we need to report quantitative results.

LED spectrum (on) and the immediately following frame with LED off, at each of two intensities.

Before we proceed to the solution, let's first understand a little bit more about what is happening. In the following animation, light impinging on the n doped photo detector region produces negative carriers, assertion of a positive voltage on the shift gate creates a channel and lowers the potential to move charge into the analog shift register. When the voltage is removed, the channel is closed and charge is trapped in the analog shift register. However some charge remains in the photo detector region.

CCD pixel mode, charge accumulates in the photodector region, application of a positive pulse to the shift gate results in charge moving into the analog shift register.

The next question is how much residual charge do we have left behind in the photodetector? Here we continue the above measurement, and graph the carry over intensity as a function of the duration of the pulse to the shift gate, at each of several voltages. We find that the asymptotic floor and the rate both generally follow voltage. The time constants are similar to or less than the 60nsec time constant in the driving circuit for this board.

Carry over as a function of the duration of the shift gate pulse at voltages from 3.6V to 5,.4V,.

We might naively think of the asymptotic residual charge after one pulse of the shift gate as following a Boltzman like dependence in voltage. The following shows that the asymptotic floor from the preceding does indeed following an exponential dependence. But we also see that we are not going to clear all of the residual charge in a single pulse of the shift gate.

Residual charge after one pulse of the shift gate follows an exp(-V/a) law.

Let's think about what happens if we repeatedly pulse the shift gate. In the first pulse we had the benefit of a large population of charge carriers to push each other out of the photodetector region. For the subsequent pulses we are in a different regime for mobile charge, and eventually we have to account for trapped charge. In the README for our github repository we derive the following equation for the readout after N pulses of the shift gate,

     S(N) = R (1−R)^(N−1) ( q_0 ​− q_Δ​ ) + q_Δ​ ⋅ e^(−γ(N−1))

where q_0 is the residual charge after the first pulse of the shift gate, R is the fraction removed at each subsequent assertion of the shift gate, q_Δ​ is the charge integrated over the period between subsequent shift gate assertions and γ is the rate for releasing charge from traps in the silicon.

In the following we graph carry-over intensity versus the number of clearing pulses at different exposure levels and periods for the clearing pulse. We see that the model fits the data very well, and indeed the thermal term is an important part of the behavior. From the fitted values we can predict that about 20 clearing pulses should reduce the residual charge to the level of dark noise.

Residual carryover as a function of the number of clearing pulses, at 4V and 5usec or 10usec period.

"Proof is in the pudding". Here is the result for a spectrum at full scale intensity followed by 20 clearing pulses. As predicted, the carry over intensity is close to dark noise.

Full scale intensity for a green LED followed by 20 clearing pulses reduces residual carry level to that of dark noise.

Concluding remarks: The above should give emphasis to the idea that instrumentation is a physics problem first. Software 'averaging' and 'offsets' will not fix a poblem that originates in charge trapping; the solution requires a strategy -- in this instance involving hardware and timing -- that respects the silicon's behavior to deliver a useful level of radiometric truth.