Skip to content

Latest commit

 

History

History
528 lines (408 loc) · 16.1 KB

known-issues.rst

File metadata and controls

528 lines (408 loc) · 16.1 KB
tocdepth:3

Known issues

This is a list of known possible issues with |hisparc| stations. For each problem some steps are given which can be followed to determine if that problem is indeed occuring on your station.

Here are filters to only show problems on a specific topic: Buffer size, CPU Load, Drive Space, EventRate, Labview Usage, Memory Usage, StorageGrowth, StorageSize, TriggerRate, Uptime.

Note

Multiple issues can be related to one topic.

Each problem described below has the following fields:

First Sign:Explaining how you will probably notice the problem.
Topic:Topic the issue is related to.
Determination:This is a small guide explaining how to make sure that the problem being described is what you are experiencing.
Solution:How to solve it.
Effects:The effects of this problem.

Software

This sections concerns itself with issues related to the |hisparc| station-software.

HiSPARC Monitor does not start

Missing directory
First Sign:

When the STARTHiSPARCSoftware program runs and the other programs (|hisparc| DAQ and Updater) start normally but the |hisparc| Monitor does not appear or closes instantly.
Topic:

EventRate, StorageGrowth, StorageSize, TriggerRate, and possibly Buffer size
Determination:
  • Look in hisparc/persistent/logs/src/ for the latest log file.
  • Check if there is a line that contains the text Error: unable to open database file.
  • Look in the hisparc/persistent/data/ directory for a hsmonitor folder.
  • If it does not exist than go to the solution, otherwise this is not the problem.
Solution:

Create the missing hsmonitor directory in hisparc/persistent/data.
Effects:

The missing directory causes the |hisparc| Monitor to be unable to store events in its SQLite database, preventing it from sending events to the Nikhef datastore. Note that the |hisparc| DAQ should be unaffected. No events should be lost, the DAQ will store events in its MySQL database until the hard disc fills up.

Hard Disc Space

To many logs
First Sign:

Issue with remaining Disc Space.
Topic:

Drive Space
Determination:
  • Look in hisparc/persistent/logs/.
  • Check the size of the src directory by right-clicking on it and choosing 'Properties'.
  • Check if this is a significant fraction of the total disc space.
Solution:

Remove all logs from the src directory except for the current one (present date in dd-mm-yyyy.log). Select all (ctrl + a) logs in hisparc/persistent/logs/src. Deselect the current one (ctrl + click). Remove them using shift + delete (to bypass the Recycle Bin)
Effects:

If the disc is full the |hisparc| daq can not store events in the database, preventing the station from storing more events.
To many updaters
First Sign:

Issue with remaining Disc Space.
Topic:

Drive Space
Determination:
  • Look in hisparc/persistent/downloads/.
  • There should be some adminUpdater_v##.zip and userUnpacker_v##.exe files there.
  • By right-clicking them you can see their file size is of the order of 100 MB.
  • If there are many they can take up some space.
Solution:

Remove all userUnpacker and adminUpdater files except the newest ones. Do this by selecting them and pressing shift + delete to remove them directly.
Effects:

If the disc is full the |hisparc| daq can not store events in the database, preventing the station from storing more events.

|hisparc| DAQ Errors

Can not connect to buffer
First Sign:Red LED in |hisparc| DAQ
Topic:
Determination:From the Start menu start odbcad32.exe. Check if the hisparc buffer is there.
Solution:
Effects:The |hisparc| DAQ will not be able to store events.
Not in DAQ Mode
First Sign:
Topic:TriggerRate
Determination:Look at the program |hispdaq|, see if the button in the middle shows 'DAQ Mode'.
Solution:Click the 'DAQ Mode' button in the |hispdaq|.
Effects:When the |hispdaq| is not in DAQ Mode it will not store triggered events.

Error in |hisparc| Monitor

Malformed HisparcII.ini
First Sign:Errors in the |hisparc| Monitor: Uncatched exception in job: need more than 1 value to unpack. Restarting...
Topic:TriggerRate
Determination:Check for blank lines in the file hisparc/persistent/configuration/HisparcII.ini.
Solution:Remove any blank lines from HisparcII.ini
Effects:Errors in the |hisparc| Monitor and no TriggerRate updates for Topic.
Time difference to large
First Sign:Errors in the |monitor|: Uncatched exception in job: invalid literal for int() with base 10: 'difference too large'. Restarting...
Topic:TriggerRate
Determination:Check for the text difference to large in hisparc/persistent/configuration/HisparcII.ini.
Solution:Check the PC time, make sure that it is set to the current time. Check the GPS settings, make sure that it is working and showing the correct GPS time.
Effects:Errors in the |hisparc| Monitor and no TriggerRate updates for Topic.
400 Bad Request
First Sign:Errors in the |monitor|: Error Uploader: .. Return code: 400
Topic:StorageSize, TriggerRate
Determination:Ensure that all required variables are being uploaded: station_id, password, data and the checksum.
Solution:Check that the station number and password are entered correctly in the configuration file: hisparc/persistent/configuration/config.ini.
Effects:No data will be uploaded.
Access denied for MySQL buffer
First Sign:Error: Buffer size: Access denied for user 'buffer'@'localhost'
Topic:Buffer size
Determination:Check if the file hisparc/user/diagnosticchecks/checks.py contains only one path to a CONFIG_INI: CONFIG_INI = "../../user/hsmonitor/data/config.ini".
Solution:Open the file hisparc/user/diagnosticchecks/checks.py in a text editor like NotePad and add the following line after the line starting with CONFIG_INI = "../..: CONFIG_INI2 = "../../user/hsmonitor/data/config-password.ini". Finally several lines later is the following code: config.read(CONFIG_INI), replace this by: config.read([CONFIG_INI, CONFIG_INI2]). Save the file.
Effects:Topic can not monitor the Buffer size.

GPS

Firmware not loaded
First Sign:No GPS appears in DSP Mon
Topic:
Determination:This only occurs with |hisparc| III electronics when their firmware is not yet loaded, which is indicated by all LEDs on the unit being on.
Solution:Start the |hisparc| DAQ to load the firmware into the |hisparc| electronics.
Effects:No GPS recognized by DSP Mon.
COM Port to high
First Sign:

The GPS date attatched to events is very inaccurate, like ~1999 or ~2019.
Topic:
Determination:
  • Open Configuration -> System -> Hardware -> Browse Devices -> Com Ports.
  • If the Com ports number higher than 32 the DSP Mon GPS program can not connect to the GPS.
Solution:

Lower the COM Port Number, by direct reassignment, use the com_port_reassign utility.
Effects:

No GPS recognized by DSP Mon, this causes data to get dates like 1999 or 2019, making the data unusable.
No antenna connected
First Sign:The GPS get no satelite signals, seen in the Satelites tab of the |hisparc| DAQ.
Topic:
Determination:Open DSP Mon and check the LED status of the Antenna Open. It will be yellow while some of the other LEDs are green. Also all Signal Values (SV) will be 0.
Solution:Check the GPS cable for kinks or cuts, also check if the antenna is still properly attached. Once fixed the SV should rise and turn green, as well as the Antenna Open status.
Effects:Makes the GPS unable to determine the correct date. Is falls back to some other date like 1999 or 2019, making the data unusable.
Time offset
First Sign:No coincidences with nearby stations.
Topic:
Determination:Open DSP Mon and check if the timing for the GPS is set to UTC or GPS, it should be GPS.
Solution:Set the GPS Timing to use GPS time.
Effects:There is a difference of ~16 seconds between GPS en UTC time, resulting in offset timestamps if the wrong setting is chosen.

Uploading

Proxy not set
First Sign:No data is uploaded, the local storage fills with events.
Topic:StorageSize
Determination:Run Diagnostics (LocalDiagnosticTool in older versions) to check if a proxy is required.
Solution:Run Diagnostics (LocalDiagnosticTool in older versions) to check Proxy settings, if it finds proxy settings for the system it can use these to configure them for Python. Press the Write Config.
Effects:

Firewall

Incoming firewall rules
First Sign:

All active Topic checks are critical.
Topic:

Host, Buffer size, CPU Load, Drive Space, Labview Usage, Memory Usage, Uptime
Determination:

All of the above Topic services are crtitical eventhough the software is running properly.
Solution:
  • Open the Windows Control Panel, go to Windows Firewall and Choose Advanced settings from the sidebar.
  • If required enter an administrator password.
  • Go the the Inbound Rules.
  • Look for the three rules that start with |hisparc|.
  • Open the Properties window for these rules, go the the Advanced tab.
  • Enable the rules for both the Private and Public profiles.
  • Next look for the rules called File and Printer Sharing (Echo Request - ICMPv4-In).
  • Enable those if they are not yet enabled and also ensure they are enabled for both Profiles.

Firewall settings

Effects:Topic can not monitor the PC status, VNC may also be blocked.
VPN blocked
First Sign:All Status indicators on Topic are CRITICAL.
Topic:Host, Buffer size, CPU Load, Drive Space, EventRate, Labview Usage, Memory Usage, StorageGrowth, StorageSize, TriggerRate, Uptime
Determination:Run Diagnostics (LocalDiagnosticTool in older versions) to check the VPN status.
Solution:Open TCP port 443 in the firewalls.
Effects:Topic will be unable to check the status of all services. Moreover, the |hisparc| support will be unable to log into the PC remotely to assist in case of problems.
Web blocked
First Sign:No data is uploaded, the local storage fills with events.
Topic:StorageSize
Determination:Try opening a website in a browser on the detector PC, preferably https://hisparc.utah.edu, if this fails then web traffic is blocked. If the browser has no problems, then look at the Proxy not set issue.
Solution:Open port 80 in the firewalls
Effects:The Uploader of the |hisparc| Monitor uses a HTTP POST Request to send data to our datastore, but this will be blocked if port 80 is closed.
.exe blocked
First Sign:An update is available but the software can not download it.
Topic:
Determination:Try opening the link to the updater, shown in the |hisparc| Updater, in a webbrowser. If this fails then .exe files are probably blocked by the network.
Solution:Allow downloading of .exe files through the firewall.
Effects:The Updater will see there is an update and quit the |hisparc| DAQ. However when the update fail it will restart the |hisparc| software, but then see there is an update and try again, it will be stuck in this loop.

Hardware

This sections concerns itself with issues related to the |hisparc| electronics and hardware.

Bad signals

Bad power supply
First Sign:|hisparc| DAQ might frequently loose the connection to the |hisparc| electronics or there will be fluctuations (sine) over the signal.
Topic:
Determination:Replace the power supply with a new one and see if the problem disappears.
Solution:Replace the power supply.
Effects:
Light leak
First Sign:Many small peaks (short pulses) in the signals in the |hisparc| DAQ, also the number of events will increase during day time (due to sunlight)
Topic:TriggerRate
Determination:Cover the detector with a light-tight blanket or foil. Now the extra peaks should disappear.
Solution:Patch the light-leaking parts with new foil/tape.
Effects:
Bad PMT base
First Sign:The current for one of the PMTs is very high (above 15 mA).
Topic:
Determination:Look in the |hisparc| DAQ at the current used by the PMTs. If this is above 15 mA something if probably wrong.
Solution:Try lowering the High Voltage on the PMT, or turning it off for a day. If that does not help it may need to be replaced, contact your cluster coordinator.
Effects:The PMT will not function properly.

No Devices Found

Connect to power
First Sign:|hisparc| DAQ is unable to connect to the |hisparc| electronics.
Topic:EventRate, StorageGrowth, StorageSize, TriggerRate
Determination:Start the |hisparc| DAQ, it will show a message that no device is found. Check if the LEDs on the |hisparc| electronics box are lit.
Solution:Connect the |hisparc| electronic box via the provided Power supply to a power outlet.
Effects:No data can be taken.