\documentclass{JAC2003}

%%
%%  Use \documentclass[boxit]{JAC2003}
%%  to draw a frame with the correct margins on the output.
%%
%%  Use \documentclass[acus]{JAC2003}
%%  for US letter paper layout
%%

\usepackage{graphicx}
\usepackage{booktabs}

%%
%%   VARIABLE HEIGHT FOR THE TITLE BOX (default 35mm)
%%

\setlength{\titleblockheight}{27mm}

\begin{document}
\title{HOW DID THE LHC ACCESS SYSTEM PERFORM IN 2009}

\author{L. Ponce\\ CERN, Geneva, Switzerland}

\maketitle

\begin{abstract}
As in 2008, the access system has been used in different configurations
 all along the 2009 year, following the machine status: shut-down 
(plus reparation), powering tests period and finally beam operation. 
The performances following the different evolutions introduced during the 
year and the issues, solved or still pending, will be reviewed during
 the presentation.  
The list of needed modifications for a more reliable and efficient use
 of the system for 2010 run will also be presented and in particular
 the effort for the material detection device (MAD). And finally,
 the current implementation of the software interlock between the access
 system and the power converters will be presented reviewed and discussed.

\end{abstract}


\section{Introduction}

After the commissioning period, the access system is now in operation. There
is no major safety problem as the LHC Access Safety System has demonstrated 
a high reliability, but the operation crew has faced, as in 2008, a lot of
 disturbances in the daily operation of the system. The problems do not have 
the same level of severity or priority according to the exploitation status
 of the machine, but are recurrent.

\section{Status and major issues}
As a detailed presentation of the status and problems, together with the
 different 
actions undertaken by the access system team have been presented several 
times to the 
LMC~\cite{LHCPerC}, 
the presentation at the Chamonix Workshop focused only on the main issues which 
had a
 strong impact on the operation efficiency of the system.

\subsection{Hardware}
A complete maintenance campaign has been done during the 2008-2009 shut-down. 
Even if
 most of the problems mentioned in the previous workshop have been solved,
 some issues
are still present and for some even with a higher occurrence.

\subsubsection{Mechanical problems}
Different kind of mechanical problems have been identified during the exploitation
 period : people blocked in the Personal Access Device (none of the doors re-opened 
in case of failure of the entry/exit procedure), door not closing properly or not 
opening properly. After 2 maintenance campaigns, most of the problem disappeared,
 except 
in some systematic location, like PM25 or UJ16, where the whole access point may 
need to be fully re-installed.

The problems are generally reported to the shift crew by the users and transmitted
 to GS/ASE team via the ODM (Ordre de Maintenance) system. This works fine for the
 intervention but the feedback after intervention could be improved to avoid 
repetition of ODM.

The impact of these problems was seen mainly during the HWC period when there is a 
high number of accesses after the long shut-down. There is no safety issue, but
 clearly an operation efficiency issue for users trying to access.

\subsubsection{Key distributor}
During the first year of operation of the system, several problems with giving back the 
safety token were experienced. The problem has been traced back to a hardware fault 
(grounding of the key distributor) combined with a software error (poor management 
of error handling). A new version of the software has been deployed during the 
shut-down together with a proper grounding of the box and the problem has been solved.

However another problem appeared along the year with faulty token slots : the pins
 supposed 
to guarantee that only the correct token can be put back in a given slot got stuck
 and as a consequence any key can be put back in the faulty slot, with the side effect 
that the proper token cannot be put back at all. The only cure to that situation 
is the exchange of the key distributor. This problem has been identified by GS/ASE as
a conceptual fault of the distributor box and a new hardware has been developed 
together with the contractor. New distributors are already available and all 
the faulty ones have been exchanged, but a general exchange should probably be planned
as the fault will very likely appear everywhere with aging of the system.
The impact on operation efficiency is seen when trying to come back from access mode to
 beam mode as at least one safety token is not seen as present.

During the 2009 exploitation period, we got two times the separation of the safety key 
used to open the PAD in the RESTRICTED mode from the token used to identify and lock the 
key in the distributor. Again, there is no safety issue  

 
\subsubsection{Drop of the patrol during standard entry}
Another important issue, which was really time consuming during the hardware commissioning 
period in powering PHASE 2, was the simultaneous opening of both PAD doors during a 
standard entry procedure, causing a loss of the SAFE status of the patrols.

The cause has been identified by GS/ASE after about 20 occurrences in different locations 
and circumstances. A new software has then been implemented and deployed in few weeks, 
just on time for beam operation.

After the intervention, some cases of patrol drops still occurred during standard passage 
through the PAD, but due to another reason, probably a contact problem on the doors.

The impact is still in the availability for powering tests or beam operation as 
the sector, and very often several sectors, have to be re-patrolled before switch back 
from access operation, even in RESTRICTED mode.

\subsection{LHC Access Safety System (LASS) status}
Only few modifications have been made on the LASS system since 2008 operation, and they
have all been successfully validated by the DSO tests.

The major improvement is the SPS-LHC Safety chain connection which has been upgraded
 to avoid the launching of false evacuation signals in point 8. 
The state of each SPS Element Important for Safety (EIS) is monitored independentely
 to allow 
the start of the evacuation siren according to the specification, i.e when 2 EIS
 are UNSAFE simultaneously.

The modification of the access sectorisation in point 5 for the needs of the Hardware 
Commissioning has been finally done with a few problems of logic in the
 LACS system. The diagnostics of the problem was done only after moving to the RESTRICTED mode
with the patrol done, so quite late after the intervention.

During the 2009 beam operation period, the beam was dumped only once because of the loss
of the patrol in the ATLAS cavern. The problem has been identified in the link between
the LASS and the ATLAS dedicated system SSA and is followed up by GS/ASE.

\subsection{Material Access Device (MAD) problem}
The problem of the personnel detection in the Material Access Device (MAD) is an on-going 
issue since 2008. A new software and a new hardware based on high resolution movement 
detection has been investigated and deployed all along the year. During the phase 2 powering
tests, only PM45 MAD access point was equipped.

The impact on the operation of the system, as well as for the users is probably the 
most painful one. Before the installation of the new software, we registered obvious 
violation of the rule. After the installation in PM45 with the first set of parameters, 
system was still too easy to violate and was anyway not deployed everywhere. So, 
for the beam operations, new compensatory measures had to be put in place for about one week: 
human guarding of the MAD by OP crew during the access period in RESTRICTED mode.

After one week of operation with the measures in place, the sensitivity of the system was 
pushed and the system deployed on all MADs, on surface and in the tunnel. The compensatory 
measures have been then lifted provided consolidation of the system will continue.

During the beam operation at the end of 2009,the system was stable, but at the beginning of 
the 2010 short shut-down, users had to face a lot of trouble to enter the material: false 
triggers 
 of the obstruction detection system due to bad image reference, micro-movements detected 
(material stabilization, melting snow, flexi...).

The system still needs a lot of development efforts and follow-up from GS/ASE to optimize
 the settings, provide a better monitoring of the system status and guarantee a fail-safe 
behaviour as requested when the compensatory measures were lifted. 
  
\subsection{LHC Access Control System ( LACS) status}
The Access Control System is the main issue for operation due to a very slow and not 
optimized application. As in 2008, the access modes were used to control activities 
during the Hardware Commissioning creating a Heavy load on the system and the operators.
We had an average of 200 keys taken per day with peak period of requests spread over
 the whole machine.

Several versions of the application have been deployed and tested during the year, but 
did not give satisfactory improvements and even led to the degradation of performance.
So, we had to roll back to previous version to get back acceptable level of stability.

\subsubsection{New features in 2009} 
The distribution sequence of the safety tokens (restricted mode keys) has been modified 
to implement a daisy chain algorithm: 
it is always the next key which is distributed to a new user, avoiding blocking of the 
access point when there is a faulty token or an inversion of tokens.

Another important improvement for operation is the deployment of the filtering of key 
requests per access point to avoid giving key to a wrong location when controlling
activities.

Furthermore,after GS/ASE spent a lot of time debugging with IT the network and routers
 related problems , most of the mis-behaviors of the system has been solved avoiding 
mixing the keys when giving them back.

\subsubsection{Remaining issues}
All these new features, requested by The Operation group gave a good improvement in the 
availability of the system for access.
However some other problems are still pending since last year without proposed solutions for the moment:
\begin{Itemize}
\item Recorder problems in few access points leading to no video availability;
\item Delays in access request transmission longer than the time-out of the request;
\item Crash of the application;
\item Quality of the audio connection, especially when cryogenic installation are ON;
\item ALICE delegation instability;
\end{Itemize}

Another issue appears this year which is the crash of the gateway between the LACS and 
the LASS. As a consequence, the change of mode are not transmitted, which might 
result in the impossibility of using the RESTRICTED mode.
The problems appeared at least 3 times. The procedure to recover from the problem has been well
 improved but the redundancy of the gateway should be sorted out as the gateway
 is critical for the availability of the system. 


Once more, none of these problems represents a safety issue, but generates a lot of 
delay in the access procedure handling, leading to very long queuing time for the user.

On the other hand, some features still need to be implemented, again to ease the use
 of the system, like improving the display of total person count in the underground area at the 
surface level,avoiding switching of video and intercom on double PAD access points.




\subsubsection{OP main requirements}
In order to compensate the slow access procedure, the operation crew needs a 
more efficient LACS application to allow multiple key distribution for a group of 
users (up to 35 persons at the same time in an access point) and to allow handling 
different access points in parallel when several groups are entering at different 
places at the same time.

Some modifications in situ are also needed to avoid people leaving with the safety token
which prevents putting back the system in beam mode when the machine is empty. Different 
solutions could be studied, e.g opening of the key distributor independently
 of which PAD is used for double PAD access points or the use of the token during 
the exit procedure to force people to have the token in hand.  


A proposal as been made by T. Hakulinen to develop in short term a multi view 
application combined with the unlinking of the interlock switch from the key 
distribution.

\section{Interlocking with the power converters}
Following the 19$^{th}$ of September, to reinforce the respect of the rules 
for access, the interlocking of the Access system with the power converters
 was raised. To date, the only communication of the Access system with CMW 
is through the signals published via TIM.
Therefore as a short term solution to help the Engineer in Charge to apply 
the procedures, a software interlock has been implemented and deployed in March 2009.

The principle is to prevent powering above the phase 1 current limit~\cite{phase1} when people are
in zone and to stop the Power Converters of the concerned areas when people are entering
 a zone.

\subsection{Implementation}
The interlock is based on the Software Interlock System 
(SIS) to generate the logic between the access conditions and the current read in the PCs
 and send commands to the power 
converters via the Power Interlock Controller (PIC). In case conditions are not met
 a global remove PC permit is sent to all the powering sectors concerned, provoking 
a slow power abort of the targeted power converters. 

\subsection{Logic used in 2009}
There is one powering permit per sector, so eight independent signals.
For each powering sector, the SIS interlock is TRUE if one of the following conditions is TRUE:
\begin{Itemize}
\item The access conditions for the sectors are TRUE (= SAFE), according to the access 
matrix~\cite{matrix},
\item The BIC loop is armed (= SAFE),
\item The measured current  in all the monitored power converters of the sectors are within 
the pre-defined phase 1 current limits~\cite{phase1},
\item The power converters are in simulation mode.
\end{Itemize}
The signal are combined in a OR logic and a global Remove Permit command is exported
 to the PIC of the concerned sector(s) when the powering OK switches from TRUE to  FALSE,
 i.e. when all the 4 previous conditions are FALSE simultaneously.

As the connection of the LASS alarms to CMW was not planned in the design, a long chain of 
different software to connect the signals is used. To provide the sum signal per powering sector, 
the signals are transmitted through the LASS PLC, the DAQ, JMS and the TIM server.
Then to publish the access conditions to the SIS, from the TIM server, the signals are going 
through JMS again, JAPC publisher, JAPC and finally, after evaluation of the logic tree in the SIS, 
a command is 
sent to the PIC which then will switch off the power converters.
Part of the software are under the responsibility of GS/ASE, the other part in BE-CO or OP groups.

The reliability of the interlock is just as good as the long chain of software can be.

\subsection{ performance}
In 2009, the interlock has been tested and validated sector by sector. The interlock always 
worked as expected, as it is designed to be fail-safe: in case one element of the chain is down, 
the sum result is force to FALSE.

In a couple of cases, the interlock rightly aborted powering tests which were launched in the 
wrong powering phase conditions.

During 2009, with the first version of the interlock, we encountered some 
availability problems. During the Hardware Commissioning period, there was one day of down-time
because of a major TIM crash. Then during beam operation, the whole LHC machine power converters 
were switched off 2 times due to an overload of the JMS broker by BLM data. The beam was 
never dumped because of the interlock, but the protection with the BIC loop status entry is 
active only when beam is present in the machine; as soon as beam is dumped, the loop opened
and if the others conditions are not met (i.e. because of missing signals) then the global
 remove permit command is sent.
\subsection{modifications}

In order to increase the reliability of the interlock during beam operation in 2010, the BIC
 loop status condition has been replaced by the reading of the LASS status in the BIS. This 
allows to ignore the sectorized access conditions when the whole machine is in BEAM ON mode 
and avoid to switch off PCs when the whole machine is empty and SAFE, even if signals are missing.

In view of keeping the interlock running for longer than initially 
expected, while waiting for a more robust solution, a possible improvement would be 
to shorten the software chain by including the JAPC publisher in the TIM server. The JMS broker 
should also be made more reliable, or separated from heavy loaded system.

For a longer term, the migration of the software interlock to a hardware interlock has to be 
well studied.
\section{Ventilation doors}
The access matrix for powering phase 2 assumed that some ventilation doors should be closed
and some opened to allow access in adjacent sectors when powering. Big efforts have been put
by GS/ASE team to equip the newly installed doors and monitor all the needed doors.
The status of each door is displayed in the CCC via the LASER alarm system and within 
a view in the TIM viewer tool used by OP-TI.

In view of the operation in 2010 and later on, the inclusion of these doors statuses in the 
software interlock (short term) or in the Safety System (longer term) has to be studied.  
\section{Some organizational issues}
During 2009 shut-down, the procedure of taking an EIS out-of-chain has been well improved, even if 
the manipulation is still delicate and risky. The management of the request is 
done via an EDMS document for a clear follow-up and summary of the status.
In parallel, the management of the access requests via the new ADI system worked far better
 than in 2009.

 
Since the first passage of the machine in RESTRICTED mode in 2008, the circulation through 
the inter-site door is an on-going discussion. A clear procedure need to be defined, keeping 
in mind that there is an issue for people follow-up in the underground areas : with the 
intersite door opened, it is 
possible then to enter in point 4 and to go almost to point 7 without being traced by the 
LACS system which can only check passage through the PAD.
The problem is further discussed in other presentations during the same session.


\section{Conclusions}
The golden period for Operation of the access system is during beam operation. Most of the
 problems mentioned become significant during massive access period. During the shut-down, 
there is a big increase of hardware problems and complains with the MAD usage. During the powering 
tests, the constraints on circulation and the long waiting time due to single passage procedure
is really painful for the users as well as for the operator of the access system.
Modifications of the access console is mandatory to increase the efficiency during the whole 
entry procedure.

An important issue to be followed-up is the faulty token which, if systematic, will need a
 replacement
of all the token distributors. 

And finally, the software interlock implemented to help applying the access restrictions 
during powering could be improved for more reliability, but will never be more reliable than 
a hardware interlock.
\section{Acknowledgments}
I would like to thank all the Operation group for the feedback and the help in the 
preparation of this presentation, especially M. Gruw\'e. I would like
 also to address a special thanks to all the access volunteers, the patrollers, 
the site managers and TI for the 
patience and availability to help managing access problems.


\begin{thebibliography}{9}   % Use for  1-9  references
%\begin{thebibliography}{99} % Use for 10-99 references

\bibitem{LHCPerC}
Presentation at the LHC Performance Committee held on 30/09/2009
\bibitem{matrix} M. Gruw\'e, Access restrictions in LHC and SPS during LHC powering Phase II, LHC-OP-OSP-0016, EDMS No. 1010617

\bibitem{phase1} Matteo Solfaroli et al: LHC-MPP-ES-0002 ver.0.2, EDMS Document No.: 1001985 "ACCESS AND POWERING CONDITIONS FOR THE SUPERCONDUCTING CIRCUITS IN LHC"

\end{thebibliography}

\end{document}
