Information

Difference between Diffuse Optical Tomography and Optical Tomography

Difference between Diffuse Optical Tomography and Optical Tomography



We are searching data for your request:

Forums and discussions:
Manuals and reference books:
Data from registers:
Wait the end of the search in all databases.
Upon completion, a link will appear to access the found materials.

What are there the differences between these two modalities?

Are there forms of OT that are not DOT?


Three-Dimensional Fluorescence Diffuse Optical Tomography Using the Adaptive Spatial Prior Approach

Traditionally, the reconstruction of fluorescence diffuse optical tomography (FDOT) images requires anatomical images as prior information. To integrate anatomical information about tumors or soft tissues into FDOT, additional equipment and procedures are required, involving co-registration, contrast-agent injection, and alignment.

Methods

Herein, we propose the adaptive spatial prior approach for reconstructing FDOT images, implemented by using an initial guess of the tumor size and location for single and multi-tumor cases. This technique is most appropriately used when FDOT is combined with limited structural information or a different structural imaging system. This method iteratively modifies the location starting from the estimated location.

Results

We conduct simulations, phantom experiments and experiments on meat tissue using the adaptive spatial prior method.

Conclusions

The results of these studies demonstrate the feasibility of the adaptive spatial prior approach for FDOT imaging.

This is a preview of subscription content, access via your institution.


Difference between Diffuse Optical Tomography and Optical Tomography - Biology

1 Department of Physics, Indian Institute of Science, Bangalore, India

2 Department of Instrumentation and Applied Physics, Indian Institute of Science, Bangalore, India

Email: * [email protected], [email protected], [email protected]

Copyright © 2013 Samir Kumar Biswas et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received October 22, 2013 revised November 22, 2013 accepted November 29, 2013

Keywords: Diffuse Optical Tomography Gauss Newton Methods Broyden and Adjoint Broyden Approaches Pseudo-Dynamic Method

Diffuse optical tomography (DOT) using near-infrared (NIR) light is a promising tool for noninvasive imaging of deep tissue. The approach is capable of reconstructing the quantitative optical parameters (absorption coefficient and scattering coefficient) of a soft tissue. The motivation for reconstructing the optical property variation is that it and, in particular, the absorption coefficient variation, can be used to diagnose different metabolic and disease states of tissue. In DOT, like any other medical imaging modality, the aim is to produce a reconstruction with good spatial resolution and in contrast with noisy measurements. The parameter recovery known as inverse problem in highly scattering biological tissues is a nonlinear and ill-posed problem and is generally solved through iterative methods. The algorithm uses a forward model to arrive at a prediction flux density at the tissue boundary. The forward model uses light transport models such as stochastic Monte Carlo simulation or deterministic methods such as radioactive transfer equation (RTE) or a simplified version of RTE namely the diffusion equation (DE). The finite element method (FEM) is used for discretizing the diffusion equation. The frequently used algorithm for solving the inverse problem is Newton-based Model based Iterative Image Reconstruction (N-MoBIIR). Many Variants of Gauss-Newton approaches are proposed for DOT reconstruction. The focuses of such developments are 1) to reduce the computational complexity 2) to improve spatial recovery and 3) to improve contrast recovery. These algorithms are 1) Hessian based MoBIIR 2) Broyden-based MoBIIR 3) adjoint Broyden-based MoBIIR and 4) pseudo-dynamic approaches.

Diffuse Optical Tomography (DOT) provides an approach to probing highly scattering media such as tissue using low-energy near infra-red light (NIR) using the boundary measurements to reconstruct images of the optical parameter map of the media. Low power (1 - 10 milliwatt) NIR laser light, modulated by 100 MHz sinusoidal signal is passed through a tissue, and the existing light intensity and phase are measured on the boundary. The predominant effects are the optical absorption and scattering. The transport of photons through a biological tissue is well established through diffusion equation [1-6] which models the propagation of light through the highly scattering turbid media.

The forward model frequently uses light transport models such as stochastic Monte Carlo simulation [7] or deterministic methods such as radiative transfer equation (RTE) [8]. Under certain conditions such as, the light transport problem can be simplified by the diffusion equation (DE) [9]. The RTE is the most appropriate model for light transport through an inhomogeneous material. The RTE has many advantages which include the possibility of modelling the light transport through an irregular tissue medium. However, it is computationally very expensive. So the DOT systems use the diffusion based approach. Gauss-Newton Method [2]is most frequently used for solving the DOT problem. The methods based on Monte-Carlo are perturbation reconstruction methods [10-12]. The numerical methods used for discretizing the DE are the finite difference method (FDM) [13], and the finite element method (FEM) [2]. Hybrid FEM models with RTE for locations close to the source and DE for others regions have also been considered [14]. The FEM discretization scheme considers that the solution region comprises many small interconnected tiny subregions and gives a piece wise approximation to the governing equation i.e. the complex partial differential equation is reduced to a set of linear or non-linear simultaneous equations. Thus the reconstruction problem is a nonlinear optimization problem where the objective function defined as the norm of the difference between the model predicted flux and the actual measurement data for a given set of optical parameters is minimized. One method of overcoming the ill-posedness is to incorporate a regularization parameter. Regularization methods replace the original ill-posed problem with a better conditioned but related one in order to diminish the effects of noise in data and produce a regularized solution to the original problem.

A discretized version of diffusion equation is solved using finite element method (FEM) for providing the forward model for photon transport. The solution of the forward problem is used for computing the Jacobian and the simultaneous equation is solved using conjugate gradient search.

In this study, we look at many approaches used for solving the DOT problem. In DOT, the number of unknowns far exceeds the number of measurements. An accurate and reliable reconstruction procedure is essential to make DOT a practically relevant diagnostic tool. The iterative methods are often used for solving this type of both nonlinear and ill-posed problems based on nonlinear optimization in order to minimize a data-model misfit functional. The algorithm comprises a two-step procedure. The first step involves propagation of light to generate the so-called ‘forward data’ or prediction data and an update procedure that uses the difference between the prediction data and measurement data for the incremental parameter distribution. For the parameter update, one often uses a variation of Newton’s method in the hope of producing the parameter update in the right direction leading to the minimization of the error functional. This involves the computation of the Jacobian of the forward light propagation equation in each iteration. The approach is termed as model based iterative image reconstruction (MoBIIR).

The DOT involves an intense computational block that iteratively recovers unknown discretized parameter vectors from partial and noisy boundary measurement data. Being ill-posed, the reconstruction problem often requires regularization to yield meaningful results. To keep the dimension of the unknown parameters vector within reasonable limits and thus ensure the inversion procedure less ill-posed and tractable, the DOT usually attempts to solve only 2-D problems, recovering 2-D cross-sections with which 3-D images could be built-up by stacking multiple 2-D planes. The most formidable difficulty in crossing over a full-blown 3D problem is the disproportionate increase in the parameter vector dimension (a typical tenfold increase) compared to the data dimension where one cannot expect an increase beyond 2 - 3 folds. This makes the reconstruction problem more severely ill-posed to the extent that the iterations are rendered intractably owing to larger null-spaces for the (discretized) system matrices. As the iteration progresses, the mismatch (, the difference between the computed and measurement value) decreases.

The main drawback of a Newton based MoBIIR algorithm (N-MoBIIR) is the computational complexity of the algorithm. The Jacobian computation in each iteration is the root cause of the high computation time. The Broyden approach is proposed to reduce the computation time by an order of magnitude. In the Broyden-based approach, Jacobian is calculated only once with uniform distribution of optical parameters to start with and then in each iteration. It is updated over the region of interest (ROI) only using a rank-1 update procedure.. The idea behind the Jacobian (J) update is the step gradient of adjoint operator at ROI that localizes the inhomogeneities. The other difficulty with MoBIIR is that it requires regularization to ease the ill-posedness of the problem. However, the selection of a regularization parameter is arbitrary. An alternative route to the above iterative solution is through introducing an artificial dynamics in the system and treating the steady-state response of the artificially evolving dynamical system as a solution. This alternative also avoids an explicit inversion of the linearized operator as in the Gauss-Newton update equation and thus helps to get away with the regularization.

The light diffusion equation in frequency domain is,

(1)

where is the photon flux, is the diffusion coefficient and is given by

(2)

and are absorption coefficient and reduced scattering coefficient respectively. The input photon is from a source of constant intensity located at. The transmitted output optical signal measured by a photomultiplier tube.

The DOT problem is represented by a non-linear operator given by where gives model predicted data over the domain and M is the computed measurement vector obtained from and.

(3)

The image reconstruction problem seeks to find a solution such that the difference between the model predicted and the experimental measurement is minimum. To minimize the error, the cost functional is minimized and the cost functional is defined as [1]

(4)

where is norm. Through Gauss-Newton and Levenberg-Marquardt [1,15,16] algorithms, the minimized form of the above equation is given as,

(5)

where is the difference between model predicted data and experimental measurement data

, and J is the Jacobian matrix which has been evaluated at each iteration of MoBIIR algorithm ( Figure 1 ). The above equation is the parameter update expression. In Equation 5, I is the identity matrix whose dimension is equal to the dimension of JJ. is regularization parameter and the order of magnitude of I should be near to that of JJ. The impact of noise and on the reconstruction is discussed in results section. The Figure 1 gives a flow chart of the approach based on Gauss Newton.

Figure 1 . Flowchart for image reconstruction by Newton’s method based MoBIIR algorithm.

2.2. Hessian Based Approach

The iterative reconstruction algorithm recovers an approximation to from the set of boundary measurements. By directly Taylor expanding Equation 3, and using a quadratic term involving Hessian, the perturbation equation can be written as,

(6)

where is the Hessian corresponding to the measurement. For d number of detectors, the above equation can be rewritten as,

(7)

The Equation 7 is the update equation obtained from the quadratic perturbation equation. The term is often observed to be diagonally dominant and can be denoted by, neglecting the off diagonal terms. Thus, through the incorporation of the second derivative term, the update equation has a generalized form with a physically consistent regularization term.

The major constraint of Newton’s method is the computationally expensive Jacobian evaluation [17,18]. The quasi-Newton methods provide an approximate Jacobian [19]. Samir et al [5] has developed an algorithm making use of Broyden’s method [17,18,20] to improve the Jacobian update operation, which happens to be the major computational bottleneck of Newton-based MoBIIR. Broyden’s method approximates the Newton direction by using an approximation of the Jacobian which is updated as iteration progresses. Broyden method uses the current estimate of the Jacobian and improves it by taking the solution of the secant equation that is a minimal modification to. For this purpose one may apply rank-one updates. We have assumed that we have a nonsingular matrix and we wish to produce an approximate through rank-1 updates [21]. The forward solution can be expressed in terms of derivatives of the forward solution using Taylor expansion as,

(8)

The Broyden’s Jacobian update equation becomes

(9)

Equation 9 is referred to as Broyden’s update equation. In Broyden’s method there is no clue about the initial Jacobian estimate [22]. The initial value of Jacobian is computed through analytical methods based on adjoint principles. It is found that since Jacobian update is only approximate, the number of iterations required by Broyden method for convergence is higher than that of gauss-Newton methods. This can be improved by re-calculating Jacobian using adjoint method when Jacobian is found to be outside the confidence domain (when MSE of the current estimate is less than MSE of the previous estimate). If the initial guess is sufficiently close to the actual optical parameter then the is sufficiently close to and the solution converges q-superlinearly to. The most notable feature of Broyden approach is that it avoids direct computation of Jacobian, thereby providing a faster algorithm for DOT reconstruction.

2.4. Adjoint Broyden Based MoBIIR

Least change secant based Adjoint Broyden [23] update method is another approach for updating the system Jacobian approximately.

The direct and adjoint tangent conditions are

and

respectively. With respect to the Frobenius norm, the least change update of a matrix to a matrix

satisfies the direct secant condition and the adjoint secant condition, for normalized directions and, and is given as [23]

(10)

where,. The rank-1 update for Jacobian update based on adjoint method is given as [5],

(11)

The Adjoint Broyden update is categorized based on the choice of. For simplicity, we consider only secant direction [23] which is defined as,

(12)

where is the step size and is estimated through line search method. The above equation has been solved in Adjoint Broyden based MoBIIR image reconstruction.

The image reconstruction flowchart using Broyden based MoBIIR is shown in Figure 2 . The Jacobian is updated through Equation 9 and Equation 11 for Broyden method and adjoint Broyden method respectively.

Figure 2 . Flowchart for image reconstruction by Broydenbased MoBIIR (Equation 9) algorithm.

2.5. Pseudo-Dynamic Approaches

Diffuse optical tomographic imaging is an ill-posed problem, and a regularization term is used in image reconstruction to overcome this limitation. Several regularization schemes have been proposed in the literature [24]. However, choosing the right regularization parameter is a tedious task. A some what regularizationinsensitive route to computing the parameter updates using the normal equations Equation 5 or Equation 7 is to introduce an artificial time variable [25,26]. Such pseudodynamical systems, typically in the form of ordinary differential equations (ODE-s), may then be integrated and the updated parameter vector recovered once either a pseudo steady-state is reached or a suitable stopping rule is applied to the evolving parameter profile (the latter being necessary if the measured data are few and noisy). Samir et al [5] have used the above approach to arrive at a DOT image reconstruction.

For the DOT problem, the pseudo-time linearized ODE-s for the Gauss-Newton’s normal equation for is given by:

(13)

where, ,

and

when we use Equation 5. For the case where the quadratic perturbation is used (Equation 7), then S is replaced by

(14)

We first consider the linear case wherein Equation 5 is used to arrive at the pseudo-dynamic system. While initiating the pseudo-time recursion for, the initial parameter vector may be taken corresponding to the background value which is assumed to be known. Equation 13 may be integrated in closed-form leading to the following pseudo-time evolution,

(15)

where and. In the ideal case, when the measured data is noise-free, the sequence has a limit point, which yields the desired reconstruction. In a practical scenario where the measured data is noisy, i.e, with being a measure of the noise ‘strength’. In this case, a stopping rule may have to be imposed so that the sequence is stopped at (

is the stopping time) with.

Figure 3 gives the reconstruction results with a single embedded inhomogeneity. Figure 3 (a) is the phantom with one inhomogeneity. The reconstructed images using Newton-based MoBIIR, Broyden-based MoBIIR and adjoint Broyden-based MoBIIR are given in (b), (c), and (d) respectively.

Figure 4 gives the performance of the algorithm. It is seen that adjoint Broyden converges faster compared to other algorithms. Figure 4 (a) shows that Newton, Broyden and adjoint Broyden methods converge at, and iterations respectively. The cross section through the center of the inhomogeneities is shown in Figure 4 (b).

Figure 5 gives the reconstruction results with two embedded inhomogeneities. Figure 5 (a) is the phantom. The reconstructed images using Newton-based MoBIIR, Broyden-based MoBIIR and adjoint Broyden-based MoBIIR are given in (b), (c), and (d).

Figure 6 gives the performance of the algorithm with two inhomogeneities. MSE of reconstructed images with two inhomogeneities is shown in Figure 6 (a). Figure 6 (b)

(a)(b)

Figure 3 . (a) Phantom with one inhomogeneity having and of size 8.2 mm at (0, 19.2). Reconstructed images using (b) Newton-based MoBIIR (c) Broyden-based MoBIIR (d) Adjoint Broyden-based MoBIIR.

(a) (b)

Figure 4 . (a) Newton, Broyden and adjoint Broyden methods. They converge at, and iterations respectively (b) Cross-section through the center of inhomogeneities for Newton, Broyden and adjoint Broyden methods.

(a)(b)

Figure 5 . (a) Original simulated phantom with two inhomogeneities The of the inhomogeneities are 0.02 and 0.015 and are at (0, �.2) and (0, 19.2) respectively Reconstructed results using (b) Newton (c) Broyden (d) adjoint Broyden method.

shows that Newton, Broyden and adjoint Broyden methods converge at, and iterations respectively. The line plot through the center of the inhomogeneities is shown in Figure 6 (c).

Figure 7 gives the reconstruction results with two embedded inhomogeneities. Figure 7 (a) is the reconstructed image by Gauss-Newton method. Figures 7(b) and (c) are the reconstructed images by linear pseudo-dynamic time marching algorithm and by non-linear (Hessian integrated) pseudo-dynamic time marching algorithm respectively.

Figure 8 analyzes the results. The line plot through the center of inhomogeneities is shown in Figure 8 (a). The variation of the Normalized Mean Square Error (MSE) with Iteration is shown in Figure 8 (b). The blue line represents Gauss-Newton’s method and green line represents pseudo dynamic time matching algorithm.

In this study, we look at many approaches used for solving the DOT problem. Like any medical image reconstruction algorithms, the main focus is to reconstruct a high resolution image with good contrast. Since the

(a)(b)(c)

Figure 6 . (a) MSE of reconstructed images with two inhomogeneities (b) Newton, Broyden and adjoint Broyden methods converge at, and iterations respectively (c) Line plot through the center of the inhomogeneities using Newton’s and proposed algorithms.

(a)(b)(c)

Figure 7 . (a) Reconstructed image by Gauss-Newton method (b) Reconstructed image by Linear pseudo-dynamic time marching algorithm (c) Reconstructed image by non-linear (Hessian integrated) pseudo-dynamic time marching algorithm

(a)(b)

Figure 8 . (a) The cross-sectional line plot through reconstructed inhomogeneity (b) The variation of the Normalized Mean Square Error (MSE) with Iteration. The blue line represents Gauss-Newton’s method and green line represents pseudo dynamic time matching algorithm.

problem is non-linear and ill-posed, the iterative methods are often used for solving this type of problems. We have summarized a few studies we undertook towards this. They are 1) Gauss-Newton based MoBIIR 2) Quadratic Gauss-Newton, Broyden-based MoBIIR 3) Adjoint Broyden based MoBIIR, and pseudo-dynamic approaches.


2. Methodology

2.1. Finite Element Solver

We consider the diffusion approximation to the radiative transfer equation [24, 25] in either steady-state, time, or frequency domain as the forward model for light transport in tissue. For steady-state problems, the stationary real-valued photon density inside the medium arising from a continuous-wave source is computed while for frequency-domain problems, the source is amplitude modulated, giving rise to a complex-valued solution of a photon density wave distribution. In time-domain problems, the source is considered a delta-pulse in time, and the measurement consists of the temporal dispersion of the transmitted signal. Given a compact domain Ω bounded by ∂Ω, the diffusion equation [26] in time and frequency domain is given by

respectively, where ω is the angular source modulation frequency, κ(r) 𠂚nd  μ a(r) are the spatially varying diffusion and absorption coefficients, respectively, where κ = [3(μ a + μ s)] 𢄡 with scattering coefficient μ s. c is the speed of light in the medium, and ϕ ,    and    ϕ ^ are the real and complex-valued photon density fields. For simplicity in the following, we use ϕ to denote either the real or complex-valued properties as appropriate.

A Robin-type boundary condition [27] applies at ∂Ω,

where q is a real or complex-valued source distribution as appropriate, ζ(n) is a boundary reflectance term incorporating the refractive index n at the tissue-air interface, and ν is the surface normal at surface point ξ. The boundary operator defining the exitance Γ through ∂Ω is given by the Dirichlet-to-Neumann map

The set of measurements y ij from a source distribution q i is obtained by integrating Γ over the measurement profiles m j(ξ) on the surface

For the time-domain problem, y ij are the temporal dispersion profiles of the received signal intensities while, for the frequency-domain problem, y ij are given by the complex exitance values, usually expressed by logarithmic amplitude ln⁡A and phase shift φ [28],

Given the set of forward data y = <y ij> of all measurements from all source distributions, (1) to (4) define the forward model f[κ, μ a] = y which maps a parameter distribution κ, μ a to measurements for a given domain geometry, modulation frequency, source distributions, and measurement profiles.

The forward model is solved numerically by using a finite element approach. A division of domain Ω into tetrahedral elements defined by N vertex nodes provides a piecewise polynomial basis for the parameters κ, μ a, and photon density ϕ. The approximate field ϕ h (r) at any point rΩ is given by interpolation of the nodal coefficients ϕ i using piecewise polynomial shape functions u i(r)

Piecewise polynomial approximations κ h , μ a h to the continuous parameters, defined by the nodal coefficients κ i, μ a,i are constructed in the same way. Applying a Galerkin approach transforms the continuous problem of (1) into an N-dimensional discrete problem of finding the nodal field values Φ = <ϕ i> at all nodes i, given the set of nodal parameters x = <κ i, μ a,i>. For the frequency-domain problem, the resulting linear system is given by


Discussion

In the past few years, OCT has been rapidly implemented into diagnosis 7,13,66 and monitoring of retinal diseases 67,68 . Currently, such measurements are widely used in humans 69,70,71 , but not routinely employed in animals (Fig. 6). Hence, this is the first report of an automated DL segmentation of vitreo-retinal and choroidal compartments in healthy cynomolgus monkeys, a species commonly used as animal models of human disease as well as for safety assessment in preclinical trials. The translation of a previously developed and reproducible ML framework in humans 24 to animals was successful. This suggests that the basic DL framework was also applicable to animals after the ML was specifically adjusted and trained on animal data.

a A high-resolution histological hematoxylin and eosin staining of a paraffin-embedded cross-section of a normal cynomolgus monkey’s eye. b Corresponding OCT B-scan from another cynomolgus monkey’s eye. Illustrated in both images are the vitreous-retina border (ILM, single arrow), internal part of choriocapillaris (CCi, double arrows), and choroid–sclera interface (CSI, arrow heads).

While ML enhanced the discovery of complicated patterns in OCT data and showed similar performance to humans 37 , there is still an unmet need for a better understanding on how ML exactly learns 72,73,74 . Typically, data are inserted into an ML environment and the results produced on the other side are often associated with a great degree of uncertainty about what is happening in between. This is referred to as ML black box (BBX). Clarifying this black box entails creating a comprehensive ML approach ideally designated for the human cognitive scale.

Thus, to solve part of this black box issue and to foster transparency, we propose an ML display concept which is designated as T-REX technique. T-REX is based on three main components or gears: ground truth generation by several independent graders, computation of Hamming distances between all graders and the machine’s predictions, and a sophisticated data visualization which is termed as neural recording (NR) of machine learning. In analogy to a mechanical gearbox, consisting of an arrangement of machine parts with various interconnected gears, we understand an ML gearbox to be composed of fine-tuned software elements which, when properly linked, should provide insight into the inner workings of the entire machine. In this sense, the ML learning process would receive a better and appropriate appreciation to determine what characteristic data the algorithm uses to make decisions. Given the overlap between neuroscience and ML, we understand by the notion of NR the registration and visual display of the predictive performance of a machine learning algorithm and human graders related to the ambiguity in the ground truth data, so that the values are presented in a comprehensive way to ML experts but are also suitable for people with a lower level of ML expertise. This is even more important to reach a larger audience so that researchers outside the ML domain who are less familiar with ML complexities can obtain a more straightforward approach to the findings.

To facilitate understanding of the Hamming distance values, visual representations of the data were conducted using MDS plots (Fig. 4) and a heatmap plot (Fig. 5). With regard to the aforementioned, this form of neural recording enables data scientists, regulators, and end users like medical doctors to better understand the impact each human grader had on the predictive performance of a trained machine learning model and thereby enhancing the understanding of a machine’s decision process. Therefore, an interesting added value of this study is that it is now possible to develop a more detailed understanding of what a CNN values in learning from each single OCT image annotation. Thus, with regard to the decision-making process of a CNN, the depth of the level of detail considered in short, the ML decision granularity was increased. The proposed T-REX methodology showed which part of the ground truth was more important: generally, graders 1 and 2 were more relevant than grader 3 because the CNN was almost always located closer to g1 and g2 (Figs. 4 and 5). Thus, g1 and g2 seem to have influenced the CNN more during learning.

Interestingly, these findings also correlate with the level of OCT expertise. Grader g1 and g2 have a higher level of expertise in OCT imaging than grader g3. So, it could be assumed that g1 and g2 generated more consistent annotations, which could have drawn the CNN predictions closer to them than to grader g3. This adaptive performance can be assumed to be directed, i.e., it seems to be “aim-oriented”. Such a mode of behavior is usually attributed to the term “intelligence”.

Overall, a good predictive performance was observed: The minor overall average per-pixel variability between the trained CNN and the human graders (1.75%) was notably lower than the inter-human variability (2.02%). The gross range of variability was congruent to previous reports 75,76 . Our results unraveled the CNN’s problem-solving skills and behavior as a form of learning a kind of robust average among all the human graders. This fact further supports the utilization of DL-based tools for the task of image segmentation, especially as the CNN performs the same task repeatedly producing the same output—independent from any physical or mental state compared to humans.

For comparing our results of the OCT segmentation to previous works, we put our results in the context of an analogous study in humans 24 . The study design differs, but still, the comparison gives insight into the CNN performance. In the study with humans 24 , a CNN was trained based on only one experienced grader, and verified with multiple human graders at three points in time. In contrast, in this study, the CNN was trained and verified with the same three human graders that labeled the images at one single point in time. In the human study, the overall inter-human variability was 2.3%, and the overall human–CNN variability was 2.0%, while the variability of three runs of the ground truth grader with the CNN was 1.6%. The range of variability was also congruent to previous reports 75,76 . As these numbers are higher than in the monkey study presented here, the actual improvements in the study design consequently might increase the performance of the presented CNN. The balancing behavior pattern during the CNN prediction, as unveiled with T-REX, reveals that in such an ML study, it is advisable to train the CNN with several graders—not just with a single gold standard expert. The proposed study design makes a CNN more robust and inherently includes an external validation 37 .

By analyzing the ML Hamming distance patterns, evidence has not only been found to support an actively balanced type of computational ML regime that can underlie any ML procedure. A similar performance was also shown in cortical circuits although of course artificial neural networks represent very rough simplifications of brain functions 77 . Dependent on the compartment, i.e., characteristic data label, the CNN judges the importance of the labels of the three human graders during training differently. For the vitreous and retina compartments, g2 and g3 produced labels relatively similarly, and g1 produced labels relatively differently. During training the CNN seems to pay more attention to the labels of the two graders who labeled similarly since the mean Hamming distance across the test set of 200 B-scans is closer to g2 and g3 than to g1 (Fig. 1). On the other hand, for the choroid and the sclera compartments, g1 and g2 labeled relatively similarly and g2 and g3 labeled relatively similarly. But the labels of g1 and g3 were relatively different. In this case, the CNN learned to make predictions that are closer to g1 and g2 than to g3. This behavior can be compared to a gear shift. Depending on the compartment, i.e. the data label, the CNN applies a different learning strategy with respect to the ground truth data. This demonstrates the importance of employing multiple independent graders for CNN training. Although this is a well-known and expected phenomenon in machine learning, it is nevertheless remarkable that this circumstance has become visually representable with this work and perceptible in such a way.

T-REX, our proposed XAI approach, can be helpful to narrow down the numerous possibilities for the development and enhancement of artificially generated knowledge for example, to select which grader provides the best opportunities for ML development or which intentional manipulations induce a deterioration in performance 73 . In particular, our T-REX analysis showed that it is necessary to study not only on the mean predictive behavior of the CNN but also to consider individual predictions on a deeper data level (e.g. each single B-scan) to transform machine learning into valuable learning. Considering only the mean predictive performance could be misleading since it would be possible for a CNN to predict certain images like human grader 1 and others like human grader 2 or 3. Analyzing the predictive performance of a CNN on individual images allows to make more precise statements about the factors that impact the learning process from ambiguous ground truth data. In general, this will enable the targeted manipulation of the ML framework in the future to document and display performance to objectively benchmark ML models and ground truth data against each other and thus improve and accelerate development. If it is better understood how the machine works, then it will also be possible to work out a set of correct premises to guide the deep neural networks in their learning and to facilitate robustness and generalization.

In order to be able to compare the ML models of different research groups, it would be ideal if they would make not only their code but also their data publicly available. Data sharing is usually restricted due to privacy of health data or data with commercial or intellectual property sensitivity. Therefore, “ML black data” exists beside ML black boxes. T-REX would be an interesting option here to generate indirect clues about the characteristics of such restricted data used so that third parties could better understand and verify the claims made.

Compared to other reports using CNNs a limit of this study could be the relatively low number of annotated ground truth data. However, the average Hamming distance between the human graders and the CNN was 0.0175 corresponding to 1.75% of pixels being labeled differently by the human graders and the CNN, respectively. This high predictive performance of the CNN was confirmed when training on the smaller ground truth data set of 800 B-Scans, which yielded similar results. This indicates that the ground truth size of 900 B-scans was sufficient to sustain the claims proposed in this study. However, it can be speculated that an even higher number of ground truth data could further improve the results. Nevertheless, the annotation of ground truth data by humans is a very time-consuming process and the current study setup appears to be an acceptable compromise between human effort and CNN predictive performance, particularly considering that the development of ML algorithms often aims at reducing the human workload.

The image quality could also have impacted the results, especially in intensely pigmented eyes due to signal loss. Moreover, there are other possible score systems than the Hamming distance and the Hamming distance does not consider how large the compartments are. In certain situations, e.g., myopia, the choroid can be much thinner than the retina, which could possibly lead to a larger difference, but that was not the scope of this study.

It is worth noting that the individual elements used in this study, i.e. U-Net, Hamming distance, MDS, and heatmap plots, might not be considered as a methodological novelty on themselves. However, the scientific originality of our work can be viewed as a unique combination of pre-existing components 78 or as a permutation of new and old information 79 : T-REX and its associated scientific discoveries in this study provide subsequent studies with a distinctive technique and a combination of knowledge not available from previous reports. In short, the appropriate conceptualization of the mentioned ML elements into the proposed framework improved the understanding of the interface between automatic computing and life sciences and therefore represents nevertheless a specific degree of originality.

Above all and despite all limits, in medicine, physicians will only use an AI system for diagnosis and monitoring of diseases if they can understand and comprehend the internal AI processing. More importantly, physicians will only make a clinical decision based on a recommendation of such an AI system if they can fully identify themselves with the AI. A subset of XAI methods aims at revealing post hoc insights into “why” a machine has taken a certain decision. While well-known post hoc approaches such as LRP or GradCAM visualize relevant regions in the input data, T-REX, our proposed XAI method, visualized and evaluated similarities between the CNN predictions and the labels of different humans that the CNN has learned from. Therefore, this study contributes to a better explainability in the application of AI, such that a resulting DL model can be better appreciated. T-REX can provide a rigorous evaluation and re-calibration tool to incorporate new DL standards. In a more general sense, it can increase the quality of explanations that are based on DL systems, which increases causability 55 . This in turn can promote safety for doctors and patients. Accordingly, the proposed post hoc XAI approach T-REX is expected to enable data scientists to model more transparent DL systems. In return, this leads to further insights into trained DL models by physicians, which utilize DL for data-supported clinical decisions.

The proposed method T-REX is not limited to semantic image segmentation in ophthalmology. In fact, it can be applied to improve the understanding of any machine learning algorithm that learns from ambiguous ground truth data. For example, T-REX could be used in the application of uncovering biases of ML prediction models in digital histopathology not only with respect to data set biases but also with respect to varying opinions of experts labeling the histopathology images 80 . In applications, where supervised ML decision models are trained to detect diseases such as Covid-19 (ref. 81 ) and experts still need to explore and agree upon specificities of the particular disease, T-REX would be helpful to visualize the ambiguity of the experts’ opinions, i.e., labels. Hence, T-REX might be especially important if the ambiguity is irresolvable meaning that domain experts disagree about the true labels, but the differences cannot be eliminated in a straightforward way. In many medical applications, the true labels cannot be verified because applying invasive procedures to patients is impossible. Therefore, methods such as T-REX, which highlight the results of the model training from ambiguous ground truth, help to improve the understanding of the objectivity of a trained model and can lead to a reduction of bias in the ground truth.

In a wider context, T-REX might yield insights into how AI algorithms make decision under uncertainty, a process very familiar to humans but so far less understood in the field of AI.


References and links

1. A. Yodh and B. Chance, “Spectroscopy and imaging with diffusing light,” Phys. Today 48:34–40 (1995). [CrossRef]

2. S. R. Arridge and J. C. Hebden, “Optical Imaging in Medicine: II. Modelling and reconstruction,” Phys. Med. Biol. 42:841–854 (1997). [CrossRef]

3. M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995). [CrossRef]

4. H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996). [CrossRef]

5. M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997). [CrossRef]

6. S. R. Arridge, “Forward and inverse problems in time-resolved infrared imaging,” SPIE Proceedings IS11:35–64 (1993).

7. A. C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging (IEEE Press, New York, 1988).

8. D. A. Boas, “A fundamental limitation of linearized algorithms for diffuse optical tomography,” Opt. Express 1:404–413 (1997). [CrossRef]

9. X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997). [CrossRef]

10. X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997). [CrossRef]

11. C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997). [CrossRef]

12. C. L. Matson, “A diffraction tomographic model of the forward problem using diffuse photon density waves,” Opt. Express 1:6–11 (1997). [CrossRef]

13. D. Boas, Diffuse Photon Probes of Structural and Dynamical Properties of Turbid Media: Theory and Biomedical Applications, A Ph.D. Dissertation in Physics, University of Pennsylvania, 1996.

14. D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994). [CrossRef]

15. P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993). [CrossRef]

16. S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995). [CrossRef]

17. R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994). [CrossRef]

18. T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992). [CrossRef]

References

  • View by:
  • Article Order
  • |
  • Year
  • |
  • Author
  • |
  • Publication
  1. A. Yodh and B. Chance, “Spectroscopy and imaging with diffusing light,” Phys. Today 48:34–40 (1995).
    [Crossref]
  2. S. R. Arridge and J. C. Hebden, “Optical Imaging in Medicine: II. Modelling and reconstruction,” Phys. Med. Biol. 42:841–854 (1997).
    [Crossref]
  3. M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
    [Crossref]
  4. H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
    [Crossref]
  5. M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997).
    [Crossref]
  6. S. R. Arridge, “Forward and inverse problems in time-resolved infrared imaging,” SPIE Proceedings IS11:35–64 (1993).
  7. A. C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging (IEEE Press, New York, 1988).
  8. D. A. Boas, “A fundamental limitation of linearized algorithms for diffuse optical tomography,” Opt. Express 1:404–413 (1997).
    [Crossref]
  9. X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
    [Crossref]
  10. X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
    [Crossref]
  11. C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
    [Crossref]
  12. C. L. Matson, “A diffraction tomographic model of the forward problem using diffuse photon density waves,” Opt. Express 1:6–11 (1997).
    [Crossref]
  13. D. Boas, Diffuse Photon Probes of Structural and Dynamical Properties of Turbid Media: Theory and Biomedical Applications, A Ph.D. Dissertation in Physics, University of Pennsylvania, 1996.
  14. D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
    [Crossref]
  15. P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993).
    [Crossref]
  16. S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995).
    [Crossref]
  17. R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
    [Crossref]
  18. T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992).
    [Crossref]

1997 (7)

S. R. Arridge and J. C. Hebden, “Optical Imaging in Medicine: II. Modelling and reconstruction,” Phys. Med. Biol. 42:841–854 (1997).
[Crossref]

D. A. Boas, “A fundamental limitation of linearized algorithms for diffuse optical tomography,” Opt. Express 1:404–413 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
[Crossref]

C. L. Matson, “A diffraction tomographic model of the forward problem using diffuse photon density waves,” Opt. Express 1:6–11 (1997).
[Crossref]

M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997).
[Crossref]

1996 (1)

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

1995 (3)

A. Yodh and B. Chance, “Spectroscopy and imaging with diffusing light,” Phys. Today 48:34–40 (1995).
[Crossref]

M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
[Crossref]

S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995).
[Crossref]

1994 (2)

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
[Crossref]

1993 (2)

P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993).
[Crossref]

S. R. Arridge, “Forward and inverse problems in time-resolved infrared imaging,” SPIE Proceedings IS11:35–64 (1993).

1992 (1)

T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992).
[Crossref]

Arridge, S. R.

S. R. Arridge and J. C. Hebden, “Optical Imaging in Medicine: II. Modelling and reconstruction,” Phys. Med. Biol. 42:841–854 (1997).
[Crossref]

S. R. Arridge, “Forward and inverse problems in time-resolved infrared imaging,” SPIE Proceedings IS11:35–64 (1993).

Boas, D.

D. Boas, Diffuse Photon Probes of Structural and Dynamical Properties of Turbid Media: Theory and Biomedical Applications, A Ph.D. Dissertation in Physics, University of Pennsylvania, 1996.

Boas, D. A.

D. A. Boas, “A fundamental limitation of linearized algorithms for diffuse optical tomography,” Opt. Express 1:404–413 (1997).
[Crossref]

M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
[Crossref]

D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
[Crossref]

Chance, B.

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
[Crossref]

A. Yodh and B. Chance, “Spectroscopy and imaging with diffusing light,” Phys. Today 48:34–40 (1995).
[Crossref]

S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995).
[Crossref]

D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
[Crossref]

Clark, N.

C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
[Crossref]

Den Outer, P. N.

P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993).
[Crossref]

Durduran, T.

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

Farrell, T. J.

T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992).
[Crossref]

Fender, J. S.

C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
[Crossref]

Feng, S.

S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995).
[Crossref]

Feng, T.

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

Frank, R. M.

M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997).
[Crossref]

Haskell, R. C.

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

Hebden, J. C.

S. R. Arridge and J. C. Hebden, “Optical Imaging in Medicine: II. Modelling and reconstruction,” Phys. Med. Biol. 42:841–854 (1997).
[Crossref]

Jiang, H.

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

Kak, A. C.

A. C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging (IEEE Press, New York, 1988).

Klibanov, M. V.

M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997).
[Crossref]

Lagendijk, A.

P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993).
[Crossref]

Li, X. D.

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

Lucas, T. R.

M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997).
[Crossref]

Matson, C. L.

C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
[Crossref]

C. L. Matson, “A diffraction tomographic model of the forward problem using diffuse photon density waves,” Opt. Express 1:6–11 (1997).
[Crossref]

McAdams, M. S.

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

McMackin, l.

C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
[Crossref]

Nieuwenhuizen, T. M.

P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993).
[Crossref]

O’Leary, M. A.

M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
[Crossref]

D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
[Crossref]

Osterberg, U. L.

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

Pattanayak, D. N.

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

Patterson, M. S.

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992).
[Crossref]

Paulsen, K. D.

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

Pogue, B. W.

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

Slaney, M.

A. C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging (IEEE Press, New York, 1988).

Svaasand, L. O.

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

Tromberg, B. J.

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

Tsay, T.

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

Wilson, B.

T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992).
[Crossref]

Yodh, A.

A. Yodh and B. Chance, “Spectroscopy and imaging with diffusing light,” Phys. Today 48:34–40 (1995).
[Crossref]

Yodh, A. G.

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
[Crossref]

D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
[Crossref]

Zeng, F.

S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995).
[Crossref]

Appl. Opt. (2)

C. L. Matson, N. Clark, l. McMackin, and J. S. Fender, “Three-dimensional tumor localization in thick tissue with the use of diffuse photon-density waves,” Appl. Opt. 36:214–220 (1997).
[Crossref]

S. Feng, F. Zeng, and B. Chance, “Photon migration in the presence of a single defect: a perturbation analysis,” Appl. Opt. 34:3826–3837 (1995).
[Crossref]

Inverse Probl. (1)

M. V. Klibanov, T. R. Lucas, and R. M. Frank, “A fast and accurate imaging algorithm in optical /diffusion tomography,” Inverse Probl. 13:1341–1361 (1997).
[Crossref]

J. Opt. Soc. Am. A (2)

H. Jiang, K. D. Paulsen, U. L. Osterberg, B. W. Pogue, and M. S. Patterson, “Optical image reconstruction using frequency-domain data: Simulations and experiments,” J. Opt. Soc. Am. A 13:253–266 (1996).
[Crossref]

P. N. den Outer, T. M. Nieuwenhuizen, and A. Lagendijk, “Location of objects in multiple-scattering media,” J. Opt. Soc. Am. A 10:1209–1218 (1993).
[Crossref]

J.Opt.Soc.of Am.A (1)

R. C. Haskell, L. O. Svaasand, T. Tsay, T. Feng, M. S. McAdams, and B. J. Tromberg, “Boundary conditions for the diffusion equation in radiative transfer,” J.Opt.Soc.of Am.A 11:2727–2741 (1994).
[Crossref]

Medical Physics (1)

T. J. Farrell, M. S. Patterson, and B. Wilson, ”A diffusion theory model of spatially resolved, steady state diffuse reflectance for the noninvasive determination of tissue optical properties in vivo,” Medical Physics 19:879–888 (1992).
[Crossref]

Opt. Express (2)

C. L. Matson, “A diffraction tomographic model of the forward problem using diffuse photon density waves,” Opt. Express 1:6–11 (1997).
[Crossref]

D. A. Boas, “A fundamental limitation of linearized algorithms for diffuse optical tomography,” Opt. Express 1:404–413 (1997).
[Crossref]

Opt. Lett. (3)

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biochemical imaging with diffuse-photon density waves,” Opt. Lett. 22:573–575 (1997).
[Crossref]

X. D. Li, T. Durduran, A. G. Yodh, B. Chance, and D. N. Pattanayak, “Diffraction tomography for biomedical imaging with diffuse photon density waves: errata,” Opt. Lett. 22:1198 (1997).
[Crossref]

M. A. O’Leary, D. A. Boas, B. Chance, and A. G. Yodh, “Experimental images of heterogeneous turbid media by frequency-domain diffusing-photon tomography,” Opt. Lett. 20:426–428 (1995).
[Crossref]

Phys. Med. Biol. (1)

S. R. Arridge and J. C. Hebden, “Optical Imaging in Medicine: II. Modelling and reconstruction,” Phys. Med. Biol. 42:841–854 (1997).
[Crossref]

Phys. Today (1)

A. Yodh and B. Chance, “Spectroscopy and imaging with diffusing light,” Phys. Today 48:34–40 (1995).
[Crossref]

Proc. Natl. Acad. Sci. USA (1)

D. A. Boas, M. A. O’Leary, B. Chance, and A. G. Yodh, “Scattering of diffuse photon density waves by spherical inhomogeneties within turbid media: analytic solution and applications,” Proc. Natl. Acad. Sci. USA 91:4887–4891 (1994).
[Crossref]

SPIE Proceedings (1)

S. R. Arridge, “Forward and inverse problems in time-resolved infrared imaging,” SPIE Proceedings IS11:35–64 (1993).

Other (2)

A. C. Kak and M. Slaney, Principles of Computerized Tomographic Imaging (IEEE Press, New York, 1988).

D. Boas, Diffuse Photon Probes of Structural and Dynamical Properties of Turbid Media: Theory and Biomedical Applications, A Ph.D. Dissertation in Physics, University of Pennsylvania, 1996.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.


Fig. 3

MTF profiles of the graphs shown in Fig. 2, illustrating the information contained as a function of spatial frequency, with LSF having highest resolution, PSF having next highest, and ESF having the lowest. Resolution is always considerably worse in the interior of the diffusing medium than at the edge near a source or detector.

Application of Resolution Testing in the Field of Diffuse Tomography

In diffuse light imaging, it has long been recognized that light follows a statistical path in which the predominant path between source and detector is a line surrounded by a banana-shaped distribution. 63 This spreading of the photon paths is induced by the inherent multiple scattering present, and decreases in width in a medium with lower scattering or increased absorption. The effect of increased absorption is somewhat counterintuitive, but generally leads to a loss of photons that have traveled farther in tissue that subsequently narrows the average path of travel. These distributions have been studied by many investigators, and specifically quantified by papers in the early 1990’s. 30, 51, 52, 56, 64, 65 Imaging of edges has not proven all that useful, as the wide spread of photons really limits the ability to visualize the edge of objects clearly, and the spatial variation in the resolution ultimately complicates the analysis.

In diffuse tomography imaging, it is easier to resolve a smooth circular heterogeneity embedded in a field than step changes. 66 This is because of the fact that heterogeneities appear as symmetrically Gaussian filtered objects in the image. Almost all papers in the field of diffuse optical imaging have focused on assessing resolution by placing point objects or line objects in the field to assess spatial resolution. 4, 30, 64, 67, 68, 69, 70, 71, 72, 73 This focus has emerged from a fundamental limitation in the field of diffuse tomography stemming from the fact that all currently used reconstruction algorithms are derived at some level from perturbation theory. The Born, Rytov, and Newton methods for minimizing an objective function are all based on perturbing an initial field to find the solution. This approach is inherently optimized for imaging point objects, and the ill-posed nature of the problem, combined with significant regularization, leads to a solution that is significantly smoother than the original test field.

Once a system or algorithm is established in its ability to recover point objects, extension to multiple objects has been a common theme however, this step is both important and problematic. The most significant problem is the nonlinear response of the measured field to multiple or extended inhomogeneities, requiring an infinite number of heterogeneity configurations to fully analyze system performance.

Perhaps the only reasonable approach to characterizing the imaging field response to multiple heterogeneities is to simulate the expected distributions of values possible in vivo and use this as the limited calibration of the system and corresponding algorithm. Even with these measures taken, it is critical to evaluate these distributions with the full range of object sizes and contrasts expected in vivo, as is discussed in Sec. 3 on contrast-detail analysis.

Analysis of Luminescence and Fluorescence Diffuse Imaging Resolution

Imaging of the minimum spatial resolution is only reasonable when an effectively infinite contrast is expected. Fluorescence protein imaging or bioluminescence imaging are two of the few situations in optical imaging in vivo where it may be reasonable to expect nearly infinite contrast, when the background emission issues might be neglected or corrected. 74, 75, 76, 77 When cells are specifically transfected or modified to express an optical signal, such as a specific organ or a tumor, the background emission in the neighboring organs should be effectively zero. In green or red fluorescent protein (GFP or RFP) imaging, the background and leakage of excitation light through the filters does provide the most significant background signal however, this can be significantly reduced when wavelength-dependent fitting or wavelength-based background subtraction is used. In bioluminescence, little real background is present in most cases, and background is often simply the dark noise in the camera or light leakage into the enclosure from the room. Thus, the spatial resolution of bioluminescence or fluorescence protein imaging in vivo can be assessed by point spread function or line spread function imaging, yet little study of this has been reported. One comprehensive paper on this issue by Troy 77 showed effective point spread functions as measured in phantoms and in vivo, using small numbers of cells to assess the minimum detectable number of photons and cells. This analysis illustrated that bioluminescence is a more sensitive imaging technique in the remission geometry, by a considerable margin, due to the decrease of fluorescent protein imaging sensitivity caused by background autofluorescence. However, recent reports of fluorescence imaging in the transmission geometry will likely be more sensitive. In most applications of fluorescence or bioluminescence, the actual resolution was not the most important parameter in distinguishing the two systems, but actually the sensitivity. Resolution of bioluminescence and fluorescence appeared to be similar, because the photon spread within a spectral window was effectively equivalent. This study and other similar studies focus on minimum contrast or signal detectable, because the issue of resolution is not governed by system constraints, but rather by the physical constraints of the light transport in tissue. Resolution limits in this regime have less to do with system design than with the depth of the object to be resolved in the tissue. While the resolution of objects at the surface of a tissue can, in principle, be as high as the diffraction limit of light (i.e., near 250 nm ) given sufficient contrast, the presence of tissue motion and the quality of the imaging system typically contribute to the real imaging resolution being lowered to typically near 1 to 2 μ m when imaging at the surface of the tissue. Again, resolution clearly degrades by orders of magnitude in just a few millimeters of depth into the tissue, due to the overwhelming presence of scattering.


3. Deep Learning Methods for Optical Tomography

Deep neural networks (DNNs) have been widely applied in computer vision, such as image classification and natural language processing. 21,37 As the deeper architecture of neural network and one approach of machine learning, DNN can be generally regarded as a function approximation method by fitting the given dataset to a certain formula with a series of parameters. In other words, it can be thought of as a mapping model between the input and the output. Actually, the reconstruction of optical imaging through random medium can be treated as two steps: first, to find the transmission model between the object (source) and the observer (detector), which is usually named as the forward problem second, to compute the desired source by measurement acquired from the observer (detector), which is usually named as the inverse problem. The forward problems are conventionally obtained mathematically according to the transmission theory, while the inverse problems are usually solved by regularization approaches, such as Tikhonov, or iterative methods, such as gradient descent.

Inspired by the principle of training to learn a model from the dataset, reconstruction algorithms with machine learning or deep learning, have attracted more and more attention. Horisaki et al. proposed a support vector regression (SVR) learning architecture to reconstruct the face objects. 38 Deep fully connected networks, as a typical architecture in deep learning, were also employed in imaging through random medium. 39 Li et al. and Sinha et al. presented several researches on introducing deep convolutional neural networks into inverse problems, respectively. 40,41

The formerly proposed deep architecture in Li’s work included down-residual blocks, up-residual blocks, convolutional blocks, and bilinear sampling blocks, of which the residual blocks are illustrated in Fig. 1. The input layer was designed for the images captured by a charge coupled device (CCD), and the detector of a lens-less system was designed to validate the proposed algorithm. It was concatenated by 1 bilinear down-sampling layer, 1 single convolution block, 5 residual blocks of convolution with down-sampling and 4 residual blocks of deconvolution with up-sampling, successively. The network was trained by back propagation algorithm with the quadratic error between input and output as the cost function. 42

Fig. 1. The difference between the residual block and densely connected block. (a) Architecture of a typical residual block and (b) Architecture of a typical 3-layer dense block.

Experiments were carried out on a free-space propagation system consisting of a laser source with spatial filtering and collimation, the phase spatial light modulator (SLM) and the detector (CCD). The distance between the CCD and SLM varies in three different values. The SLM was modulated by characters randomly selected from MNIST handwritten digit database 43 and then the CCD captured the lights transmitted through the SLM and the blurred images were obtained as the input of the proposed DNN architecture. The training set consisted of 10,000 samples. The test set included 1000 digits excluding the sample used to train the network of the MNIST, 15 Chinese characters, 26 English letters, and 19 Arab characters. Mean-squared error (MSE) of the ground truth and the reconstructed results was utilized as the evaluation metric. The capacity of DNN-based approach, in reconstruction images through a free propagation system, was illustrated by the visible results. The well-trained network performed better when the distance between CCD and SLM was closer.

Different from the architecture for free propagation imaging system described above, a so-called “IDiffNet” for the inverse problem of imaging through diffuse medium was proposed in 2018. 41 Employing densely connected convolutional network, the “IDiffNet” consisted of a dilated convolutional layer with pooling, 6 dense and downsampling transition blocks, a dense block, 6 dense and upsampling transition blocks, an upsampling block and a convolutional layer with Rectified Linear Unit (ReLU).

The difference between dense block and the formerly mentioned residual block can be illustrated in Fig. 1 and formulated by 44

More skip connections were also employed in “IDiffNet”. Another variant of “IDiffNet” was that it utilized negative Pearson correlation coefficient (NPCC) as the loss function to replace the conventional MAE strategy because NPCC performed better for spatially sparse objects and strong scattering conditions. 45

Experiments with one of two different diffusers and training inputs, which were extracted from one of three databases (Faces-LFW, 46 ImageNet, 47 and MNIST 43 ) for each time, were carried out to validate the proposed method. The training size was 10,000 each time. 450 samples, which were collected from both the same database with the training dataset and other databases, were used for testing. “IDiffNet” with MAE as the loss function were utilized as the comparison of the NPCC-loss function strategy. Experiments results indicated that different scattering severities required different training datasets. In strong scattering conditions, training with a relatively constrained dataset with strong sparsity would outperform that training with generic datasets. The tradeoff was similar in choosing the loss function: NPCC was more beneficial than MAE when the scattering was strong and the object was sparse while it was slightly worse when the training dataset was collected from a generic database.

Li’s works, in actuality, indicated the feasibility and capacity of learning-based approach, in particular deep learning-based approach, in reconstruction of imaging through random medium. Since the forward problems and the inverse problems of biomedical optical imaging systems are various and more complex than the aforementioned systems, particular studies for reconstructions which are based on learning methods and focus on different modalities, such as DOT, FMT, BLT, and PAT, have been presented recently.

3.1. Model-based methods for inverse problems

DOT is generally used to investigate the optical absorption and scattering parameters of the biological tissues, indicating the anomalies. The forward model can be conventionally generated from light propagation models such as diffusion equation. 48 Actually, the forward model of DOT together with FMT and other modalities can be written in the form

Ignoring the reflections of light propagation through the medium, Kamilov et al. proposed a multi-layer artificial neural network 49 which employed beam propagation method (BPM) to model scattering process in three-dimensional (3D) phase objects. 50,51 Heterogeneous medium here was divided into several slices along the propagation direction. As shown in Fig. 2, each slice was evenly sampled along the coordinate ( x , y ) with the sample interval δ and represented by each circle on each layer of the BPM structure. Summations of the complex amplitude of the signals converging to each circle were multiplied by e j 2 π Δ n z δ z ∕ λ , with δ z denoting the sample interval along the propagation direction, Δ n ( x , y , z ) denoting the perturbation of 3D index in the medium at coordinate ( x , y , z ) , and λ denoting the wavelength, respectively. Straight line between each two circles denoted the multiplication of the output of the corresponding unit and the discretized Fresnel diffraction kernel. 52 The BPM model was represented by a multi-layer architecture and trained by error back propagation algorithm.

Fig. 2. Schematic of BPM network. 49 Each circle, which can be regarded as the node of a layer of the network, denotes the phase modulation to be learned from training the network by minimizing the error between the output of the last layer and the experimental measurement. The propagation of incident light is along the Z direction.

Experiments of imaging polystyrene beads and HeLa and hTERT-RPE1 cells via a holographic system were implanted to validate the proposed architecture. During the experiments, the imaging object was sliced into 420 layers along the 30 μ m length propagation distance and the transverse sampling interval was set to 72 nm. The training set consisted of 80 samples which were collected by illuminating the sample at 80 distinct angles. The network was initialized with the standard filtered back projection reconstruction algorithm or nonzero constants to investigate the effect of initialization. Inverse Radon transform reconstruction was employed as a comparison. The performances of the proposed BPM network algorithm and the compared algorithm were evaluated visually.

In summary, Kamilov’s work proposed a BPM network to obtain the distribution of the optical coefficients in heterogeneous medium in microscopy. The weight matrix obtained by training the network with several iterations was directly related to the desired result.

In macroscopic DOT, Yoo et al. proposed an encoder–decoder convolutional structure deep neural network to inverse the Lippmann–Schwinger integral equation for photon migration. 53,54

The network consisted of three parts: a fully connected layer mapping the measured data to voxel domain, a pair of encoder CNN layers, and an intermediate convolutional layer. Hyperbolic tangent function (tanh) was employed as the activation functions for the first three layers and ReLU was adopted for the last convolutional layer.

Experiments were carried out via imaging polypropylene phantoms and tumor-bearing mouse, respectively. The training set was generated by a FEM solver. 1,500 samples were obtained by varying the size of heterogeneous anomaly in a range of 2–13 mm in radius and by changing the optical properties of the heterogeneous background in the biologically relevant range. MSE and Adam optimizer were employed as the loss function and the optimizing strategy, respectively. 1000 of the 1500 samples were treated as the training set and the rest were treated as the validation set. Rytov iterative method was developed to be the comparison method. 55 It should be emphasized that the network was trained merely by the simulated numerical samples. The experimental results demonstrated visibly that the trained network performed well and could accurately reconstruct the anomalies without iterative procedure or linear approximation.

The approach proposed in Yoo’s research was actually an end-to-end architecture for optical tomography. The encoder–decoder CNN block was designed strongly according to the Lippmann–Schwinger integral equation, which was different from the end-to-end methods discussed later.

Another typical model-based learning approach was proposed to solve the limit-viewed inverse problem in PAT. 56 First, some theories of PAT should be depicted. The forward problem of PAT can be formulated as

The so-called “deep gradient descent (DGD)” algorithm in Hauptmann’s work focused on learning how to update x k + 1 at each iteration. For the ( k + 1 ) th iteration, the parameter θ k of a simple network structure G θ k was trained for updating x k by

Two strategies were proposed to train the DGD architecture: the first strategy was used to train the network as a whole with the predefined maximum iteration k max , while the second strategy was used to train the network parameters at each iteration sequentially. There was a tradeoff between the two approaches. The first approach offered optimal results after k max iterations while suffering from repeated evaluations of system matrix and the adjoint. The second approach had advantage in decoupling the computation of gradient and training process, providing an upper bound of training error. However, the second approach could not obtain minimal training error, failing to achieve the optimal result. Alternatively, the second approach could be employed as the initialization for the first approach.

Simulated samples, as the training set, were synthesized by preprocessing data x true extracted from the publicly available ELCAP Public Lung Image Database 57 with the following operation :

Furthermore, the DGD algorithm was updated by transfer training the network on the weakly regularized total variation (TV) reconstructions from the fully sampled data x TV and y .

Conventional iterative method which used a U-net as post-processing, 58 direct reconstruction which used limit-view data by TV method, and direct reconstruction which used fully sampled data by TV method, were employed as comparison to DGD approach and the updated DGD approach. Results of updated DGD were visibly comparable with the fully sampled TV results and quantitatively outperform others, according to evaluation metrics which included peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

3.2. Post-processing methods to refine reconstructions with deep learning

Due to the highly scattering photons in the turbid medium, the inverse problems of optical tomography are ill-conditioned and suffer from ill-posedness. As a result, it is difficult to obtain accurate quantities and locations of the objects. Reconstructions are commonly flawed by artifacts, leading to the bias estimations for the ground truths. Utilizing deep neural networks as the post-processing, two-stage methods are put forward to reduce the errors between reconstructed results and the ground truth. 59

Take Long’s work for instance, a two-stage method generally employed a conventional iterative approach to obtain the preliminary reconstruction (the fluorochrome distribution in FMT) from the measured data and adopted a CNN to deblur the preliminary reconstruction. 59 In Long’s research for FMT, the conventional iterative approach utilized in the first stage was depth-dependent Tikhonov regularization, which had been widely used in solving the inverse problem. 60 The second stage consisted of three 3D convolutional layers, one max pooling layer, two 3D convolutional layers, and two fully connected layers, successively.

Numerical experiments were carried out by employing Monte Carlo (MC)-based forward-adjoint method to simulate the photon propagation. 61 Phantoms with one target and two targets were designed to enrich the datasets. The first stage was implemented with the regular procedure. In the second stage, both simulated data and reconstructed data were selected to balance the proportion of negative and positive samples. The key point of this two-stage method is that the CNN has been employed to learn whether a patch, including several voxels obtained from discretizing the whole medium into voxels and nodes, is on the boundary of the fluorescence target to be reconstructed.

Although the performance of this two-stage method in refining the FMT reconstructions has been verified, the shortage still exists: the proposed CNN method can only work on the ideal situation that the fluorescence targets to be reconstructed are homogeneous. In other words, the proposed two-stage method in Ref. 59 will fail if the concentration of the targets varies, which is common in real experiments. Nonetheless, it provides a novel idea to deblur the reconstruction results with CNN.

As mentioned formerly in Sec. 3.1, U-nets have also been applied as post-processing to reduce the reconstruction error. Besides, Antholzer et al. applied a typical U-net to map a reconstruction with under sampling artifact, which was generated by conventional linear method, to a reconstruction image with removed artifact. 62 The proposed network was trained on simulated samples generated by numerical experiments. The advantage of this U-net post-processing method is that the total computation time of the two stages is only 20 ms, much lesser than 25 s of the conventional TV method.

3.3. End-to-end methods with deep learning

Non-model algorithms mentioned in the first few paragraphs of Sec. 3 indicate that end-to-end reconstructions by deep learning methods for optical imaging or tomography are feasible. Actually, several studies on how to map the measured data acquired at the detector to the desired result in absence of forward model have been presented.

An architecture named “Net-FLICS” (Network for Fluorescence Lifetime Imaging with Compressive Sensing) was proposed by Yao et al. to provide an end-to-end solution for reconstructions of both fluorescence intensity and lifetime. 63

As shown in Fig. 3, Net-FLICS was multi-output structured, consisting of a shared convolutional segment, an intensity image reconstruction branch, and a lifetime image reconstruction branch. The first segment, which consisted of one convolutional layer and six residual blocks, was shared by reconstruction tasks of both intensity and lifetime, filtering the measured sequences mainly along the pattern orientation. The tensor data was transposed to be ready for filtering along the temporal orientation. Split data was fed to the intensity image reconstruction branch and lifetime image reconstruction branch, respectively. In the intensity reconstruction branch, reshaped data was filtered by four two-dimensional (2D) residual blocks, each of which included three convolutional layers. In the lifetime reconstruction branch, one-dimensional (1D) convolutions were kept in four blocks and then the tensor data was reshaped and the structures were changed into 2D blocks to fit photon intensity curves in order to obtain the lifetime. Residual connections were utilized for both constructing and connecting the residual blocks. At the end of both branches, three convolutional layers were established for final image filtering.

Fig. 3. Architecture of Net-FLICS. 63 The upper section includes the shared convolutional segment and intensity image reconstruction branch (inside the orange dotted rectangle), while the lower section includes the lifetime image reconstruction branch (inside the purple dotted rectangle). The detail of nD ( n = 1 , 2 ) residual block is shown at the left bottom, respectively.

To evaluate the proposed Net-FLICS, compressed data was acquired on the compressive sensing single-pixel imaging system. 64,65 Binary images from the MNIST database were processed to generate the simulated images with various intensities and lifetimes, respectively. To augment the dataset, each image from MNIST was processed to generate six images: the original one, rotated by 90 ∘ , rotated by 180 ∘ , rotated by 270 ∘ , flipping up/down, and flipping left/right. 32,000 samples were employed as the training set and 8000 samples were the test set. Experiments with phantom containing three letters “RPI” filled with two dyes, AF750 for “ R ” and “ I ”, and HITCI for “ P ”, were also employed to validate the Net-FLICS. Compared to the TVL3 + lifetime fitting method, 66 the Net-FLICS could achieve higher reconstruction quality for the HITCT dyed letter “ P ” with the reconstructed lifetime 0.97 ns, which was the same as the ground truth, but failed in achieving the desired reconstruction for AF750 dyed letters “ R ” and “ I ”. The authors attributed the mismatch to that the patterns generated by DMD were not perfectly binary, which differed from that utilized to simulate the training set.

A multi-layer perceptron (MLP)-based inverse problem simulation (IPS) method was proposed by Gao et al. to obtain an end-to-end reconstruction for BLT. 67 The IPS architecture, which constructed the mapping from the measured data at the surface of biological tissue to the distribution of bioluminescence inside the tissue, consisted of 1 input layer, 4 hidden layers, and 1 output layer. According to Gao’s work, the IPS could be regarded as a simulation to the widely used iterative shrinkage threshold (IST) method in solving the inverse problems. Coupled samples, including both photon intensity on the surface and the internal bioluminescent source, were generated by utilizing MC method with the so-called “standard mesh” to simulate the photon migration. Samples with multiple source were generated by synthesizing samples with different located single source. The barycenter error (BCE) between the reconstructed source and the ground truth was utilized to quantitatively evaluate the results in the simulation. The IPS network was validated by evaluating it in terms of positioning accuracy, robustness, performance in the dual-source reconstruction, and performance in the reconstruction of in vivo tumor-bearing mouse model.

Cai et al. proposed another end-to-end, deep learning-based algorithm for reconstruction of quantitative photoacoustic imaging (QPAI). 68 The proposed deep neural network was called “ResU-net”, a combination of U-net and Res-net. The differences between the “ResU-net” and the basic form of U-net was that the former employed residual blocks including three convolutional layers and batch normalization between each two blocks of contraction branch and expansion branch. The channel size was kept invariant in each residual block. Simulated samples with variant illumination wavelengths, different distributions of targets, and different chromophore concentrations were generated by a numerical forward model. Three simulated experiments including a case which used a digital mouse 69 were carried out to validate the “ResU-net” for end-to-end reconstruction of QPAI, which confirmed the performance in reconstruction accuracy for chromophore concentration and the robustness for variant optical properties and geometry.

Recently, our group proposed an end-to-end algorithm for FMT reconstruction with a deep neural network framework. 70 It established a nonlinear mapping between the internal fluorochrome distribution and the boundary measurements. It was a typical 3D encoder-decoder architecture with 6 4 × 6 4 × 2 4 inputs and 6 4 × 6 4 × 1 6 outputs, as well as multiple layers of spatial convolution and deconvolution operators. 10,000 simulation samples with variable experimental settings were used as the training set while both simulation samples and phantom samples were used for validation. Results of both simulated experiments and phantom experiments demonstrated the great improvement in FMT reconstruction compared to conventional methods. The proposed algorithm can be regarded as the first 3D encoder–decoder DNN-based algorithm for FMT reconstruction with comprehensive experimental validations.

At least, an end-to-end reconstruction method requires the employed neural network to have capacities to simulate inverse problem perfectly and reduce reconstruction errors. Hence, the architecture of deep neural network and the training set should be chosen properly to avoid possible misleading effects caused by improper design.

The most dramatic advantage of the end-to-end methods is that it takes negligible time to obtain the desired result with a well-trained network although the training process may be time consuming.

In summary, all these three branches of deep learning-based methods for optical tomography are validated to be effective by a large number of experiments. The comparison is shown in Table 2. A model-based deep learning method is difficult to define. In some cases, the models which need to be simulated by deep learning framework are photon propagation models or certain parts of propagation models. While in other cases, methods with deep learning framework may focus mainly on how to model the updating of weighted matrixes. The advantage of utilizing deep learning frameworks as post-processes after obtaining the reconstructions is that some existing deep learning frameworks, such as U-net, can be adapted conveniently to the post-processes with tiny changes. However, end-to-end methods with deep learning may take place of the other methods in the future for the reason that it changes the typical principle of reconstruction for optical tomography. Final results can be obtained directly under the premise that well-designed deep learning frameworks are well trained. And of course, there are no requirements to hypothesize photon migration models except that the training sets are obtained from simulations.

Table 2. Comparison for three categories of deep learning-based methods.


Combined fDOT and microCT imaging

One hour after ICG injection or 24 hours after Angiostamp800 injection, anesthetized mice (isoflurane/air: 2%) were placed in a homemade mobile animal holder for bimodal microCT/fDOT imaging. Three-dimensional fluorescence imaging was performed on the liver using the previously described fDOT system [3]. MicroCT imaging was performed in line using the vivaCT40 system (Scanco Medical AG, Switzerland), whose acquisition parameters were 55 keV energy, 177 µA intensity, 300 ms integration time and 80 µm voxel isotropic resolution. MicroCT- and 3D-fluorescence-reconstructed volumes were merged using ImageJ software. The total fluorescence signal in the volume of interest (liver) was quantified and expressed as RLU.


INTRODUCTION

Diffuse optical tomography (DOT) is a new imaging modality with potential applications in functional imaging of the brain and breast cancer detection [1-5]. This imaging technique seeks to recover the optical parameters of tissue from boundary measurements of transmitted near-infrared or visible light. Instrumentation for optical tomography system is relatively less expensive and is portable for the clinical settings[1]. It has been proved that DOT system can provide a viable alternative to current available functional imaging systems such as functional magnetic resonance imaging[1].

A typical DOT system often consists of a light source (lasers, white light), illuminating the biological tissue from the surface at different source positions in succession[1]. The photons which propagate through tissue are then collected at multiple detector positions on the tissue surface[1]. Three measurement schemes are used for these measurements: time domain, frequency domain and continuous wave (cw). Of these three measurement types, the cw method is the simplest and least expensive, and can provide fastest data acquisition and greatest signal-to-noise level[6].

One of the applications of cw DOT system is the light dosimetry for interstitial prostate photodynamic therapy (PDT). The effectiveness of PDT treatment largely depends on the number of photons absorbed by the photosensitizers located in the tumor tissue [7]. Thus the light and photosensitizer dosimetry are essential for PDT treatments [8]. In our prostate PDT protocol, optical properties of prostate are first determined before the treatment, so that a real-time modeling and monitoring of photons deposition in the prostate can be achieved. Prostate optical properties are determined via an interstitial DOT system where light sources and detectors are interstitially inserted in the prostate tissue [9].

In this study, we present 2D and 3D inverse models that can recover the heterogeneous optical properties of the prostate tissue. A 2D-3D hybrid model is also presented. These models are built in the COMSOL Multiphysics / MATLab environment, where the partial differential equations are solved in COMSOL Multiphysics and the heterogeneous equation coefficients are updated via Levenberg-Marquardt algorithm written in MATLab. We have validated these proposed models by reconstructing the optical properties of the 2D, 3Dmathematical phantoms with the numerically simulated data. We demonstrated that the 2D-3D hybrid algorithm outperforms the 2D and 3D algorithm in terms of accuracy/computation cost ratio.


High Resolution Diffuse Optical Tomography using Short Range Indirect Subsurface Imaging

Diffuse optical tomography (DOT) with line-scanned camera and line scanning MEMS projector (left) compared with traditional DOT with point source-detector pairs (right). Both arrangements capture short range indirect subsurface scattered light but our approach is more efficient and recovers the medium (bottom row) at much higher resolution. (click the figure for full image, including the result comparison from traditional DOT and the proposed method)

Generation of short range indirect images for a small (left) and a large (right) pixel to illumination distance. The scene consists of three cylinders embedded in a scattering medium, click the figure for the full image.

Visualization of phase function for different pixel to illumination line distance in y-z plane (top row), and x-y plane (bottom row). S and D represents the illumination line and pixel location respectively. As the pixel to illumination line distance increases, the photons tend to travel deeper into the scattering medium but leads to reduced number of photons reaching the pixel, thereby reducing the signal-to-noise ratio.

Experiment setup and calibration to compensate the laser-mirror misalignment and non-linearity of MEMS. The device is mounted vertically above the sample container, with no cover above the scattering medium. Due to misalignment, the incident laser beam onto the MEMS mirror will not be perpendicular to the mirror surface and align with the MEMS rotation center Due to non-linearity of MEMS mechanics, the input control signal and degrees of rotation are not linearly related. Click the figure for the full image.

Real data images and results for multiple objects inclusions. The scattering medium is skim milk with no or little water dilution. Click the figure for the full image

Video

We show the working scheme illustration and the reconstruction results in the video

Where does the convolution in the forward model come from?
It comes from: (1) the usage of scanning illumination and camera exporue lines to replace the source-detector pairs in traditional DOT (2) the isotropic light propagation direction distribution within dense scattering medium. For (2), the assumption holds true for dense scattering medium such as human skin. We test the robustness of our method against the failure of this assumption in the paper. Our method works for a wide range of scattering coefficients of the medium. Please refer to Section 6.1 for more details.

What do the phase function and kernel depend on ?
They depend on the source-detector (for traditional DOT using point sources and detectors) or the separation between the illumination and exporue lines (for our method), and the scattering and absorption coefficients of the homogeneous medium surrounding the heterogeneities.

Given the line separation, how to get the phase kernel? In other words, how to get the absorption and scattering coefficients for the homogeneous medium ?
We optimize the absorption and scattering coefficients of the homogeneous medium by solving an inverse problem for the homogeneous region. Theoretically the image intensities in a 1 x 1 area (a point) within the homogeneous region is enough to solve for the coefficients. We take a homogeneous region and calculate the mean values over the region to get the set of intensities to reduce noise. The homogeneous region can be either seleted manually, or automatically as the regions with low spatial contrast in all short-range indirect images.

In the experiment, what is the maximal separation between the line illumination and camera exposure line on the surface ?
In our experiment, the separation is up to around 10mm on the surface. The maximal separation setting depends on the sensed depth and data capture time budget. For larger separation, deeper heterogeneity can be sensed and reconstructed. But the signal is weaker due to more scattering and absorption, so we need to use longer camera exposures hence more time budget.

The diameter of the laser spans several pixels in the image taken with a small FoV camera. Is this considered ?
Yes. We pre-calibrate the 1D profile across the laser line illumination, and take that into account in the forward model as a spatial convolution.

What is the black spot in the center of the short-range indirect images ?
The black region is the artifact caused by the protection glass in front of the MEMS mirror we are using. Part of the laser light is reflected directly by the protection glass without being manipulated by the MEMS mirror. In addition, there is inter-reflection between the glass and MEMS mirror. As a result, in the raw image, there is a bright spot in the middle of image. This causes an issue especially for highly scattering medium since the region of the bright spot will be much larger than the laser spot itself due to subsurface scatterings.

To ameliorate this issue, we subtract the "dark" image from the captured images, where the "dark" image is captured with the laser pointing outside the sample and FoV for the camera. However, because the inter-reflection between the glass and the MEMS mirror also depends on the MEMS mirror orientation, there is still artifact (the black region) in the center of the image after we apply the simple trick. A more straightforward and promising way to get rid of this artifact is to remove the protective glass. But that should be done in a clean room to avoid the contamination of the MEMS mirror.

Is the method robust to the failure of the assumption in traditional DOT that the scattering coefficients of the heterogeneities and the surrounding homogeneous are close?
Yes. We test the algorithm with different scattering coefficients for the heterogeneities and fixed scattering coefficient for the surrounding, using Monte Carlo simulated data. The performance does not degrade if the scattering coefficient of the heterogeneity varies. Please refer to Section 6.1 in the paper for more details.

The specifications of the MEMS ?
We use the Mirrorcle MEMS development kit. The scanning actuator is with the gimbal-less two-axis design. MEMS mirror is 1.2 mm in diameter, with operational angular range from -5 to 5 degrees in both x and y axis. We use the point-to-point (quasi-static) mode in order to project vector graphs.

Related projects

Homogeneous Codes for Energy-Efficient Illumination and Imaging
Matt O'Toole, Supreeth Achar, Srinivasa G. Narasimhan and Kiriakos N. Kutulakos
ACM Transactions on Graphics (SIGGRAPH) 2015

Acquiring and Characterizing Plane-to-Ray Indirect Light Transport
Hiroyuki Kubo, Suren Jayasuriya, Takafumi Iwaguchi, Takuya Funatomi, Yasuhiro Mukaigawa and Srinivasa G. Narasimhan
IEEE International Conference on Computational Photography (ICCP) 2018