Service status updates

Subscribe via RSS | Twitter: @SRCFstatus

See also our Nagios automated test output.

Emergency maintenance, 2014-04-08

Due to a critical vulnerability in OpenSSL reported a few hours ago, most Linux systems which use SSL or TLS in any form can be made to disclose private data.

All SRCF servers will be undergoing maintenance to patch this issue tonight.  In most cases this will involve a reboot.

We apologise for the disruption caused but urgent action is required to protect users’ data.  (If you have Linux systems elsewhere should ensure that these are patched as well, even if they are not servers.)

0 notes

Outgoing mail delays

Since 22:17 on 26th February, some outgoing mail from the SRCF has been experiencing delays (in some cases, of several hours).  This is due to connectivity problems with the University’s central mail servers.

We are working with the University Computing Service to investigate and resolve the problem.  Meanwhile, apologies for the disruption.

Update 19:45: we deployed a workaround at 18:20; there should be no further delays.

The cause was an interoperability bug in the TLS implementations (leading to “Bad record MAC” errors and aborted SMTP connections) between our mail server and the University’s; previously these connections generally hadn’t used TLS at all until the University Computing Service enabled it last night.

0 notes

IPv6 unavailable on pip, 06:58-08:21

Due to an erroneous Ubuntu update triggering a long-standing Linux kernel misfeature, the main SRCF shell/web server (pip) lost IPv6 connectivity at 06:58.  This was repaired at 08:21.

IPv4 connectivity was unaffected so few people are likely to have noticed, but if it did affect you (e.g. long-standing IPv6 connections dropped, poor performance for applications which don’t support RFC 6555 (Happy Eyeballs), no reachability from IPv6-only hosts) then we apologise for the inconvenience.

0 notes

Email delays, 2013-08-16

Due to a spam incident on Thursday evening, mail leaving the SRCF servers is currently being severely rate-limited by the University mail servers and we therefore have a backlog of undelivered mail.

Mail sent from the SRCF server or forwarded through the SRCF is likely to be delayed by several hours until at least mid-morning on Friday.

Apologies for any inconvenience caused.

0 notes

Disk problem on main SRCF server (pip.srcf.net), 2013-07-23 18:31

Regrettably the main SRCF server, pip, just suffered a disk problem and is now unavailable until repairs have been completed.  Further information will be posted here as it becomes available.

Update 19:10: the disk problem does not appear to have been too catastrophic.  It is possible that a small amount of data has been corrupted, but the vast majority is intact.  Contact the sysadmins if you need assistance dealing with corrupted data.  I expect to be able to bring the server back online soon, barring further problems.

Update 20:34: service restored.  Report any remaining problems to the sysadmins.

Apologies for any inconvenience caused by this unplanned outage.

0 notes

Urgent maintenance midnight tonight

Tonight, soon after midnight, there will be a short disruption to most SRCF services as we will be installing an urgent software patch on our servers.

This will involve a reboot, so please make sure that you save your work and exit any applications you may have running either on the shell service (pip) or on the desktop service (cyclone).

Apologies for the short notice; this patch is important for the continued safety and security of the facility.

1 note

Server maintenance tonight

Tonight we will be replacing and upgrading hardware in the SRCF virtualised cluster.  This means that, once again, there will be an interruption to the following services:

  • IRC
  • Usenet news
  • Desktop

These services will be shut down for anything from half an hour to multiple hours each whilst their data is copied onto the new platform.

Arrangements have been made to maintain a basic IRC service during this maintenance; those of you wanting to remain on IRC may connect your IRC clients to drought.srcf.net.  Make sure to change them back to irc.srcf.net tomorrow after the outage.

The outage will not affect the IRC server on zeus, but that will be netsplit from drought.  If you don’t know what zeus is, ignore this and just use drought.srcf.net.

0 notes

Server reboots, 23:00 today

As per Andrew’s earlier announcement, we will be rebooting a few servers this evening from 23:00 in order to install security patches.

This will affect:

  • IRC service (flood)
  • Usenet news service (flame)
  • Desktop service (cyclone)

Connections to these services will fail for a few minutes each.  Anything left running on the Desktop service will be terminated.

This maintenance will not affect the shell or web services (pip).

0 notes

Electrical maintenance work completed without downtime

Although our electrical supply was interrupted for a brief period today as expected, we survived on battery power and the maintenance was completed without any downtime for our services.

0 notes

Possible disruption due to electrical work, 2013-02-23

On Saturday the building which houses the SRCF servers will be affected by electrical work.  This work will start at 07:30 and continue throughout the day, and will involve short interruptions to the SRCF’s power supply.

The SRCF has UPS (battery backup) systems to power the servers during short electrical outages—indeed we have an additional UPS arriving today to increase capacity and resilience—however it is still possible that the power disruption(s) will last longer than the batteries.  We may have to shut down some or all servers at short notice.

Apologies for any inconvenience caused.

0 notes