bath/parts/experiments.tex

%! TEX root = ../thesis.tex
\chapter{Experiments}
Now that the theoretical model is complete, experiments can be done to start checking that model and get results to use for in system components.

\section{Experimental Setup}
But before diving into the measurements ashort tour of both experimental setups.
THe first setup was used during the calibration phase, while the secon setup was used for creating the regulation model.

\subsection{Part 1}

    To calibrate a Powerit a setup is required, that can sweep the input voltage, as well as draw different current from the PowerIt (see \autoref{fig:expsetup1}).
    For that a setup with a bench power supply an electronic load and an external voltmeter are used.
    Additionally a STM32-Discovery board and a RaspberryPi microcomputer were connected to flash new firmware onto the PowerIt.
    \begin{figure}[H]
        \centering
        \includegraphics[width=.35\columnwidth]{./pics/setup_ps_el_mm.jpg}
        \includegraphics[width=.45\columnwidth]{./pics/poweritv1_test_setup_corr.jpg}
        \caption{%
            Photographs of the first experimental setup.
            On the left side visible are a Keithley K2100 voltmeter (top), a PS9080 bench power supply (middle) and a EL9000 electronic load (bottom).
            On the right side visible are a PowerIt with connected STM32-Discovery board (left), and Raspberry PI (top).
            Also in the picture is  the power supply connection (cables at top of PowerIt).%
        }%
        \label{fig:expsetup1}
    \end{figure}

    To now calibrate the board the bench supply could be controlled, to sweep through a voltage range, or in a similar fashio the electronic load could sweep through different current draw scenarios.
    This setup includes cable connections to voltmeter, power supply and load as to be able to control them with a piece of software.

\subsection{Part 2}

    To obtain the required measurements for creating a regulation model the second setup was used (\autoref{fig:expsetup2}).

    \begin{figure}[H]
        \centering
        \includegraphics[height=.55\columnwidth]{tikz/setup2.pdf}
        \hspace{1.5cm}
        \includegraphics[height=.53\columnwidth]{pics/setup_pw_back_2.png}
        \caption{%
            Photographs of the second experimental setup.
            In this setup the Wafer system assembly was used.
            This module has a height and length of 50cm and a width of 15cm.
            The left side shows the back side of the assembly.
            Here are the PowerIt (1), CURE (3) and AnaB (2) boards mounted, as well as a RaspberryPi (4) and a STM32-Discovery (5).
            The right side shows the empty front side of the MainPCB and the wafer heatsink.%
        }%
        \label{fig:expsetup2}
    \end{figure}
    
    This setup is similar to a BrainScaleS wafer module as it exists inside the system.
    But in contrast to these systems there are no FPGAs, AuxPwr or FCP boards (reference~\cite{waferembedding},fig 2.2)
    The MainPCB has the PowerWafer embedded and is also connected to 8 CURE boards, 2 AnaBs and a PowerIt.

\section{Characterization}

The first experiments to run are the characterization of hardware behavior. These will then result in a PowerIt Calibration, which later then can be used as basis for creating a regulation method.

\subsection{Sampling Time}
    First up was selecting an optimal number of cycles for which the adc will probe a to it connected pin, like described in \autoref{sec:adc}.

    In this case the uncalibrated measurement of input voltage was taken as example, and repeated with each of the possible 8 settings.

    To be able to compare a reference voltage measurement was taken with an external Voltmeter\footnote{Keithley K2100}.
    The resulting errors, from a set Voltage, can be seen in figures \ref{sampleticks1}

    \begin{figure}
        \centering
        \hspace*{-.175\columnwidth}
        \includegraphics[width=1.3\columnwidth]{./data/m04_cycledepends/cycledepends_20180529.pdf}
        \caption{Top: input difference from set voltage vs set voltage for different possible scaler values.
        Bottom: gain error of the linear fitted curves vs set scaler value (May 29th 2018, $\approx$32\si\degree C)}%
        \label{sampleticks1}
    \end{figure}

    \autoref{sampleticks1} contains the absolute error of the measured voltage compared to the theoretical, set input voltages. therefore the reference measurements (yellow), taken with an external Voltmeter, are not at 0.
    Also shown are the calculated gain errors, in case of all 8 settings.

    Important to note is the relative error in only the 0th case, here the \verb|cycleTime|-Setting was set to 0 and therefore the smallest available sampling time of 3 Ticks. This excludes 0 a possible value to use. All other measurements are within error margin of each other, and because a smaller time frame is preferred, the best value to use is 1, resulting in a measure time of 15 Ticks. 

    % TODO: move to appendix
    %\begin{figure}[H]
    %    \centering
    %    \hspace*{-.175\columnwidth}
    %    \includegraphics[width=1.3\columnwidth]{./data/m04_cycledepends/cycledepends_20180530.pdf}
    %    \caption{plotted difference from set input voltage, and fitted linearly, May 30th 2018, $\approx$25\si\degree C}
    %    \label{sampleticks2}
    %\end{figure}

\subsection{Voltages}

    Now that a sample time is chosen, it is possible to proceed with the voltage calibration measurements.
    Note, that measurements are expected to be less accurate, the more components are contained in their respective measurement circuit. Because small errors will accumulate and in e.g. the case of 48V's be amplified by a factor of 8.

    \subsubsection{48V Input}
    
        When looking at calibrating the input voltage (\autoref{v48_precalib}), we can clearly see a relatively constant offset of $\approx$1V.
        In \autoref{v48_precalib} a polynomial fit of 2nd degree\footnote{A Fit of second degree will be used in the complete calibration process} is done and its coefficients extracted (\autoref{pitdb}, line 9).
        These coefficients not only show an offset, but also some deviation in the incline and curve from the default values.

        \begin{figure}[H]
            \centering
            \vspace{-1cm}
            \hspace*{-.16\columnwidth}
            \includegraphics[width=1.3\columnwidth]{../pitstop/20180824/calib_v48.pdf}
            \vspace{-1cm}
            \caption{
                Calibration of \SI{48}{\volt} input voltage.
                Plotted are measured and reference vs the calculated vin voltage.
                The Calibration sweeps from \SIrange{43.2}{52.8}{\volt}.
                The fit is of second degree and its inverse are the to use calibration coefficients.
                (fit:{\(
                    \pyval{poly48v2}V_\text{IN}^2+\pyval{poly48v1}V_\text{IN}+\pyval{poly48v0}=V_\text{MONITOR\_48V}
                \)})}%
            \label{v48_precalib}
        \end{figure}

    \subsubsection{9.6V Output}

        The 9.6V Calibration, in contrast, shows only a slight deviation of the internal values and the reference measurement, which results in a list of coefficients (\autoref{
            %TODO: 
        }, line 7), very similar to those set in the theoretical defaults.
        
        \begin{figure}[H]
            \centering
            \vspace{-1cm}
            \hspace*{-.16\columnwidth}
            \includegraphics[width=1.3\columnwidth]{../pitstop/20180824/calib_v10.pdf}
            \vspace{-1cm}
            \caption{%
                Calibration of \SI{9.6}{\volt} input voltage.
                Plotted are measured and reference vs the calculated vin voltage.
                The Calibration sweeps from \SIrange{43.2}{52.8}{\volt}, and the supply mudules divide that into \SIrange{8.64}{10.56}{\volt}.
                The fit is of second degree and its inverse are the to use calibration coefficients.
                (fit:{\(
                    \pyval{poly10v2}V_\text{10V}^2+\pyval{poly10v1}V_\text{10V}+\pyval{poly1v0}=V_\text{MONITOR\_10V}
                \)})}%
            \label{v10_precalib}
        \end{figure}

        This small difference is explained by the simple voltage division used as our circuitry, and no amplification, as for the input voltage circuit.

    \subsubsection{1.8V Output}

        The last Voltage to calibrate is divided into two domains, one for supplying the analog circuitry inside the wafer system, and one for the digital side. Each deliver between 1.5%TODO: value!!
        and 2.022V and each is settable by its own circuit (both as in \autoref{fig:gen18v}).

        \begin{figure}[H]
            \centering
            \vspace{-1cm}
            \hspace*{-.15\columnwidth}
            \includegraphics[width=1.3\columnwidth]{./data/m03_poticalib/adccalib_02.pdf}
            \caption{Calibration: analog 1.8V Output voltage, plotted are external measurement and internal values vs set resistance at the BCU Voltage Module.}%
            \label{fig:v18_precalib}
        \end{figure}

        Visualized in \autoref{fig:v18_precalib} is the analog domains calibration, showing nearly no difference in board and reference measurements. Mostly due to direct connection between created voltage and the STM-Chips pin.

\subsection{Currents}
    
    With now calibrated Voltages, the next step is to measure the behavior of the current measuring circuits.
    Note that the 9.6V Output does in fact not have a include circuit for measuring its current draw, and that this number will be obtainable from all other (calibrated) measurements.

    \subsubsection{48V Input}
        This experiment will calibrate the 48V input current.
        In it the current drawn by the PowerIt sweeps over a range from \SIrange{0}{20}{\ampere}.

        In \autoref{fig:calib48i} quite a gap between observed and measured values can be seen.
        This is most likely a gain error, which would result in a error in \(m_2\), as observed.
        And the fitted curve has the following parameters:
        
        \begin{align}
            V_\text{MONITOR\_48I} =&~m_0 + m_1\cdot I_\text{IN} + m_2\cdot I_\text{IN}^2\\
            m_0 =&~\pyval{poly48i0}\nonumber\\
            m_1 =&~\pyval{poly48i1}\nonumber\\
            m_2 =&~\pyval{poly48i2}\nonumber
        \end{align}

        from which the inverse will used for calibration inside the PowerIt.

        \begin{figure}[H]
            \centering
            \vspace{-1cm}
            \hspace*{-.16\columnwidth}
            \includegraphics[width=1.3\columnwidth]{../pitstop/20180824/calib_i48.pdf}
            \vspace{-1cm}
            \caption{%
                Calibration of \SI{48}{\volt} input current.
                Plotted are measured and reference current vs the calculated pin voltage.
                The Calibration sweeps over \SIrange{0}{20}{\ampere}.
                The fit is of second degree and its inverse are the to use calibration coefficients.
            }%
            \label{fig:calib48i}
        \end{figure}

    \subsubsection{1.8V Output}

        For the calibration experiment of both 1.8V output currents, the current draw ranged from \SIrange{0}{90}{\ampere}.
        Observed were the values in \autoref{precalib18i}.
        Visible is a different incline of internal measurement and reference.

        \begin{figure}[H]
            \centering
            \vspace{-1cm}
            \hspace*{-.165\columnwidth}
            \includegraphics[width=1.3\columnwidth]{../pitstop/20180824/calib_i18.pdf}
            \vspace{-1cm}
            \caption{%
                Calibration of \SI{1.8}{\volt} output current for both digital and analog.
                Plotted are measured and reference current vs the calculated pin voltage.
                The Calibration sweeps over \SIrange{0}{90}{\ampere}.
                The fits are of second degree and their inverse are the to use calibration coefficients.
            }%
            \label{precalib18i}
        \end{figure}
        
        The fitted curve for the analog side were:
        
        \begin{align}
            V_\text{MONITOR\_1I8\_ANA} =&~m_0 + m_1 \cdot I_\text{1.8V, ana} + m_2 \cdot I_\text{1.8V, ana}^2\\
            m_0 =&~\pyval{poly18iana0}\nonumber\\
            m_1 =&~\pyval{poly18iana1}\nonumber\\
            m_2 =&~\pyval{poly18iana2}\nonumber
        \end{align}

        , while the digital side had quite similar values of: 
        
        \begin{align}
            V_\text{MONITOR\_1I8\_DIGI} =&~m_0 + m_1 \cdot I_\text{1.8V, digi} + m_2 \cdot I_\text{1.8V, digi}^2\\
            m_0 =&~\pyval{poly18idigi0}\nonumber\\
            m_1 =&~\pyval{poly18idigi1}\nonumber\\
            m_2 =&~\pyval{poly18idigi2}\nonumber
        \end{align}

        This also show, that both parts are so similar in bahavior, that a single sides observations would have sufficed.

\section{1.8V Regulation}

As Described beforehand the Output Voltages for both analog and digital can be adjusted to some degree and therefore we can compensate for the dropoff occurring between PowerIt Output Terminals and Reticles.

To run any test with the PowerWafer, the patterns in \autoref{fig:wpattern} were used.
Tere are two reasons for that, firstly these patterns distribute the current draw in a regular fashion as to distribute the load between the connectors. 
Secondly, when powering Reticles all of the energy is converted into heat, via the ohmic resistors.

\begin{figure}[H]
    \centering
    \includegraphics[width=\columnwidth]{./data/theory/wpattern.pdf}
    \caption{Used regular patterns for current tests on PowerWafer}%
    \label{fig:wpattern}
\end{figure}

And although the copper heat sink and fans (see \autoref{fig:expsetup2}), should be able to handle this heat in theory, when grouping together reticles and powering them, the dissipation does not suffice.
The internal temperature probes (between heatsink and wafer) register well above \SI{50}{\celsius}, when grouping 3 or more reticles. 

\subsection{Characterization of Dropoff}

Wanting to observe and characterize the voltage drop, first the connections between PowerIt and Reticles can be measured with the in \autoref{fig:retmodel} described connections, which in actuality are the PowerIT Terminal and corresponding analog readout pin on a Analog readout board.

To use the PowerWafer for testing one of the patterns in \autoref{fig:wpattern} will be used, each pattern has a approximate current draw of 120A and will distribute heat and draw per terminal evenly.

In \autoref{1v8dip} a single reticles (\#40) voltage drop for different Current Draws is visualized.
A relatively linear trend and residuals of a trigonometric behavior can be observed (most likely the result of the inaccurately measurable current draw, which here is done inside the PowerIt).

\begin{figure}[H]
    \centering
    \vspace*{-1.5cm}
    \hspace*{-.16\columnwidth}
    \includegraphics[width=1.3\columnwidth]{../pitstop/20180824/ret_vdip.pdf}
    \vspace{-1cm}
    \caption{Voltage drop observed between PowerIt and HICANN, each point represents a state after enabling additional Reticles on the PowerWafer (right upper wafer in \autoref{fig:wpattern})}
    \label{1v8dip}
\end{figure}

Here a Voltage Drop vs. Current draw of the wafer shows a linear behavior and therefore can be regulated on basis of the current measurement done by on board Measurement circuit.

\subsection{Numerical-Correction (Regulation)}

The initial idea, to approach the correction of this dropoff is a Numerical: the SWRM (\autoref{sec:swrm}) and its corresponding Equations can be applied here.
\autoref{eq:fullreg}, which maps the measured output current to a corresponding potentiometer setting, requires the Dropoff to be linear, which was observed.

To apply this approach, two assumptions need to be made:
\begin{itemize}
    \item all reticles have the same current draw (already not accurate, see \autoref{1v8dip})
    \item all reticles experience the same voltage drop (as observed for reticle 40)
\end{itemize}

and the following four values are required, before a regulation can be attempted:
\begin{itemize}
    \item \(I_{ret}\), the current draw of a single reticle,
    \item \(R_0\), the resistance between PowerIt and FET,
    \item \(R_1\), the resistance of a single Reticle

    \item \(V_{off}\), the wanted Voltage at a Reticle
\end{itemize}

To get a representative value of \(I_{ret}\) for use in the SWRM, the mean of all reticles current draw was taken (\autoref{fig:ihist}):

\begin{align}
    \pyval{iretmeancorr}
\end{align}

\begin{figure}[H]
    \centering
    \vspace{-1cm}
    \hspace*{-.15\columnwidth}
    \includegraphics[width=1.3\columnwidth]{../pitstop/20180819/reticle_ihist.pdf}
    \caption{Distribution of analog current draw for all reticles on the PowerWafer (which were possible to measure \(\rightarrow\) \autoref{sec:pitfalls})}%
    \label{fig:ihist}
\end{figure}~\\
The \autoref{fig:ihist} was obtained by measuring the increase in current draw for each reticle, for each of the 4 patterns (\autoref{fig:wpattern}).\\\\
To obtain \(R_0\), the pattern in \autoref{fig:wafer-ret5} was used to take measurements for both the Neighborhood as well as the Farthest Reticles.

\begin{figure}[H]
    \centering
    \hspace*{.1\columnwidth}
    \includegraphics[width=.6\columnwidth]{../pitstop/processing/neighborhoood_5.pdf}
    \vspace{-1cm}
    \caption{Reticles used to determine correlation between distance and Voltage Drop}%
    \label{fig:wafer-ret5}
\end{figure}

\begin{figure}[H]
    \centering
    \vspace*{-1cm}
    \hspace*{-.15\columnwidth}
    \includegraphics[width=1.3\columnwidth]{../pitstop/20180820/reticle_corr}
    \caption{Voltage drop vs current for both Reticles in direct neighborhood and farthest possible Reticles}%
    \label{fig:ret5corr}
\end{figure}

From \autoref{fig:ret5corr} it is possible to see that the distance between reticles that are used gives different behavior of the voltage drop. Both Inclines happen to be the extreme cases, while either being completely uncorrelated, the case for farthest Reticles, or being directly correlated by their distance, here observable for the neighboring Reticles.

Therefore we obtain two values for \(R_0\):

    \begin{align}
        \pyval{r0_from_neighbor}\\
        \pyval{r0_from_farthest}
    \end{align}

from the same measurement it is also possible to extract \(R_1\) by extrapolating to 0, which results in:

    \begin{align}
        \pyval{r1_from_neighbor}\\
        \pyval{r1_from_farthest}
    \end{align}

here the values obtained are within error margin of each other.

So applying these Values, the following behavior for regulation can be visualized:

\begin{figure}[H]
    \centering
    \hspace*{-.16\columnwidth}
    \includegraphics[width=1.3\columnwidth]{./data/theory/reg.pdf}
    \caption{Possible \(P_{val}\) curves after SWRM, dotted lines represent not achievable values}%
    \label{fig:regswrm}
\end{figure}

The in \autoref{fig:regswrm} visualized values show the theoretical \(P_{val}\) for the corresponding Current, while all dotted parts depict the values which would be needed to achieve full correction at the Reticle level. 
Note that the 1.8V regulation, should fail at about 80A of current draw.

Now that the SWRM is applicable, what about the DWRM, which removes the assumption of a equal voltage drop per reticle, applying an offset to the initially observed voltage of each reticle.

To account for that, the voltage drop per reticle, in a single reticle power state, was observed:

\begin{figure}[H]
    \centering
    \vspace*{-1cm}
    \hspace*{-.15\columnwidth}
    \includegraphics[width=1.3\columnwidth]{../pitstop/20180820/reticle_vdiphist.pdf}
    \caption{Initially observed voltage drop, red values are ignored for corrected mean}%
    \label{fig:vdiphist}
\end{figure}

and a mean of:
\begin{align}
    \pyval{vdipmeancorr}
\end{align}

can be observed.

\autoref{fig:wrdist} shows how those Voltages are Distributed over the complete PowerWafer.
All white Reticles are not measurable, and those colored in Red and Yellow are the outliers in \autoref{fig:vdiphist}. Note that in a deployed, working, Wafer System inside BrainScaleS the middle two Reticles (19 \& 28) are not used and also give grounds to ignoring the outliers.

This results in a distribution, which when combined with the spread of \(R_0\) from \autoref{fig:ret5corr}, gives an approximate range for all reticles voltage drop at a given current draw (\autoref{fig:vrange}).
The last values that were measured, came from the CURE boards connected to each reticle.
The voltages obtined from these boards, can be compare to the manually obtained voltages vie AnaB pins.
In \autoref{fig:vcure40} these voltages are visualized, in comparison to the AnaB voltages.

\begin{figure}[H]
    \centering
    \vspace{-1cm}
    \includegraphics[width=.7\columnwidth]{../pitstop/20180825/reticel_rdist.pdf}
    \vspace{-.5cm}
    \caption{\(V_\text{drop}\) distribution over full Power Wafer; White have no measurement; Red an Orange are marked red in \autoref{fig:vdiphist}}%
    \label{fig:wrdist}
\end{figure}

\begin{figure}[H]
    \centering
    \vspace{-1cm}
    \includegraphics[width=.9\columnwidth]{../pitstop/20180825/reticle_vcure.pdf}
    \vspace{-.7cm}
    \caption{%
        Qualitative comparison of AnaB and CURe measured voltage drop.
        Colors indicate their respective relative values for each measurement.
        White reticles have no measurement. 
    }%
    \label{fig:vcure40}
\end{figure}

This comparison shows no discernable relation between both measurements.
This could be the result of not correctly calibrated CURE boards, or at least hints to some problem with the measurement.