Old System News

milou2 rebooted August 28

Tue, 29 Aug 2017 07:16:00 GMT

milou2 rebooted on Monday 2017-08-28 at 19:51

milou2 rebooted August 19

Mon, 21 Aug 2017 07:06:00 GMT

milou2 rebooted on Saturday 2017-08-19.

Intelmpi

Fri, 16 Jun 2017 08:12:00 GMT

Intelmpi performance issues

Intelmpi

Fri, 16 Jun 2017 07:28:00 GMT

Intelmpi performance issues.

Issues with X11 on milou (X11Forwarding) -- SOLVED

Mon, 29 May 2017 13:36:00 GMT

We have observed and several users have reported issues with running X11 applications on Milou. We are investigating it.

milou2 and milou-b rebooted

Mon, 29 May 2017 13:34:00 GMT

The login nodes milou2.uppmax.uu.se and milou-b.uppmax.uu.se were rebooted 15:00 today (29th of May) due to some issues with the kernel NFS module.

Cooling stop at 17.00 hours the 23rd of May -- CANCELLED

Tue, 16 May 2017 12:31:00 GMT

Issues with certain project volumes for milou/pica 20170515 and onwards.

Mon, 15 May 2017 12:43:00 GMT

Some project volumes on pica are very heavily loaded and slow/next to unusable for interactive use. We're doing what we can to resolve this but can not promise any set time for when things will behave as normal again.

UPDATE: We've had some continuing issues with this due to some nodes not realizing when resources behave better, we're working on these issues but this may have caused disturbances like failed jobs or missing output.

Support may be slow May 11th and 12th due to conference

Wed, 10 May 2017 13:29:00 GMT

The UPPMAX system group hosts the spring 'SONC' conference where administrators from all SNIC-centers meet and discuss how to improve our centers. With many UPPMAX adminstrators being out of office during the conference (Thursday 11th and Friday 12th) the support will likely be less responsive.

slurm disturbance on milou 2017-05-10

Wed, 10 May 2017 10:28:00 GMT

Due to a misconfiguration active on a certain number of nodes around 12AM today, some jobs that were launched on milou could not start.

If you have jobs that were victims of this, they will likely show up as completed although with a very short run time (a few seconds).

Disturbances in Slurm today Tuesday -- finished

Tue, 09 May 2017 12:21:00 GMT

Maintenance window Wednesday 2017-05-03 -- finished

Wed, 03 May 2017 05:56:00 GMT

Slurm problems on Rackham -- fixed

Fri, 28 Apr 2017 13:51:00 GMT

Intel license server not responding --fixed

Tue, 25 Apr 2017 14:39:00 GMT

We have gotten reports that the Intel license server is not responding. We are investigating it. This might manifest itself with hangs or freezes during compilations.

Problem "Invalid account or account/partition..." --solved

Mon, 24 Apr 2017 13:10:00 GMT

We have identified a problem with the Slurm account database. If you just got added or created a new project you might get the following message when scheduling jobs "Invalid account or account/partition...". It affects primarily Rackham and Milou.

Problem with Slurm on Milou -- fixed

Tue, 18 Apr 2017 08:04:00 GMT

Interrupts in Slurm service on Rackham -- fixed

Tue, 11 Apr 2017 09:50:00 GMT

Bianca's storage system Castor has problems -- fixed

Mon, 10 Apr 2017 16:50:00 GMT

Resetting your password from the homepage is not working --fixed

Thu, 06 Apr 2017 12:28:00 GMT

Resetting your password from this page is currently not working. If you need to reset your password please contact support@uppmax.uu.se

Update 2017-04-18: This issue should now be fixed.

Funk-accounts and new certificates

Thu, 06 Apr 2017 10:23:00 GMT

Some of the shared funk-accounts used on Irma and Milou might stop working due to the IP-address change.

Maintenance window Wednesday 2017-04-05 -- finished

Wed, 05 Apr 2017 07:43:00 GMT

Smog will be decommissioned on Wednesday 5th of April

Mon, 03 Apr 2017 10:14:00 GMT

Smog will be decommissioned on Wednesday 5th of April. As previously mentioned the SNIC Cloud Team is currently working on bringing up a new cloud to replace Smog and join the other two regions in the SNIC Science Cloud project.

For questions ,please contact support@cloud.snic.se (and not the UPPMAX support queues).

Rackham2, one of Rackham's login nodes, got into problems -- now fixed

Wed, 29 Mar 2017 13:11:00 GMT

Maintenance window for Bianca Wednesday 2017-03-22 -- finished

Wed, 22 Mar 2017 08:20:00 GMT

Problem with file permissions in certain projects

Fri, 17 Mar 2017 12:03:00 GMT

Poor performance using Intel MPI on Rackham

Mon, 13 Mar 2017 11:00:00 GMT

We have idenfied performance issues when using Intel MPI on Rackham. In some cases you see a 10x slowdown (or worse) using Intel MPI compared to Open MPI. We are investigating this issue and hope to have it solved soon. For now, please use Open MPI.

Fixed: "Project p123456 may not run jobs on this cluster (rackham)"

Thu, 09 Mar 2017 14:30:00 GMT

An issue exist on Rackham affecting projects of the form "p123456". The projects are not allowed to run due to the monthly core allocation incorrectly being set to 0 hours. We are investigating why this happens.

Update 2017-03-10: The issue should now be fixed.

Rackham is now open for all users

Wed, 08 Mar 2017 07:48:00 GMT

All active Tintin projects (exception Tintin-Fysast1, please see below) have been migrated to Rackham. All UPPMAX users should now have access to Rackham.

Rackham will soon be open for all users

Mon, 06 Mar 2017 11:05:00 GMT

Many Tintin users have missed that Rackham will replace Tintin. We are currently migrating all projects from Tintin to Rackham and when this is done, all users will get access to Rackham. We will announce this per email and on our homepage.

Maintenance window Wednesday 2017-03-01 -- finished

Wed, 01 Mar 2017 08:11:00 GMT

Today we decommission Tintin

Wed, 01 Mar 2017 08:03:00 GMT

1st of March 2017 is the day we decommission Tintin. It will be replaced by the Rackham cluster. All projects on Tintin will be moved to our new Rackham cluster.

Creation of new UPPMAX user accounts will be delayed

Fri, 24 Feb 2017 12:48:00 GMT

Delayed approval of Account Requests -- fixed

Wed, 22 Feb 2017 13:05:00 GMT

We have identified a problem with the UPPMAX Account Request which unfortunately causes some delay before you can login to UPPMAX. We hope to complete the registration this week. You do not have to resubmit your Accounts.

Problem sending in support tickets using support@uppmax.uu.se -- fixed

Fri, 17 Feb 2017 08:56:00 GMT

There is currently a problem sending in support tickets to support@uppmax.uu.se. We are investigating and hope to have it fixed soon.

Rackham is now available

Wed, 01 Feb 2017 12:29:00 GMT

We are happy to announce that UPPMAX's cluster Rackham is now available!

Downtime due to power outage

Tue, 24 Jan 2017 07:48:00 GMT

Milou, Tintin, and Fysast-1 are back in production. Bianca is back in test production. Still working on Smog.

Milou2 now back again

Wed, 18 Jan 2017 13:03:00 GMT

The degraded RAID now fixed

Milou-f rebooted tuesday afternoon

Tue, 17 Jan 2017 16:11:00 GMT

Lustre file system problem

Milou1 rebooted (tuesday 14:00)

Tue, 17 Jan 2017 12:58:00 GMT

Totally inresponsible. Lustre file system problem (wich will be decommissioned tomorrow....)

Milou1 rebooted -- now with limited number of inodes on /scratch (/tmp)

Tue, 17 Jan 2017 08:29:00 GMT

We have now quota on the number of files in /scratch (/tmp).

100000 is maximum (per user). If you need more you have to use a compute node.

Gulo (including glob directory) decommissioned January 18

Mon, 16 Jan 2017 14:58:00 GMT

Milou2 down for reinstallation 13:50 (now waiting for spare parts)

Mon, 16 Jan 2017 10:53:00 GMT

Milou2 hasn't worked well for a while. We will give him a fresch restart.

Milou1 rebooted Thursday am11.00

Thu, 12 Jan 2017 11:46:00 GMT

Milou1 rebooted Thursday am11.

Problems with lustre file system.

Fysast1 down Wednesday

Wed, 11 Jan 2017 14:45:00 GMT

Fysast1 down Wednesday before lunch.

One power supply broken and the fuses for half the cluster was blown.

Milou1 rebooted Wednesday am11.00

Wed, 11 Jan 2017 10:33:00 GMT

Milou1 rebooted Wednesday am11.00

Problems with lustre file system

Maintenance window on Irma Wednesday 2017-01-11 -- FINISHED

Wed, 11 Jan 2017 09:29:00 GMT

Maintnenace window on Mosler/topolino Wednesday Jan 11 - FINISHED

Tue, 10 Jan 2017 10:18:00 GMT

We have a maintenance window coming up on January 11 from 9:00. Due to physical work, we need to shut down the system during the maintenance window this time so jobs won't run.

We will also likely be required to rebuild virtual nodes and will probably lose information about queued jobs.

Update 21:10: Maintenance is now finished and the system should be available again.

Poor performance on Milou and Tintin

Thu, 05 Jan 2017 13:55:00 GMT

Maintenance window Wednesday 2017-01-04 -- FINISHED

Wed, 04 Jan 2017 07:19:00 GMT

Milou2 rebooted friday morning

Fri, 23 Dec 2016 06:57:00 GMT

Milou2 rebooted at 06:01 due to a problem with the lustre filesystem

Reduced staff availability over the coming holidays

Tue, 20 Dec 2016 13:45:00 GMT

Milou-f rebooted friday night

Mon, 19 Dec 2016 07:16:00 GMT

Problems with lustre file system

Milou2 rebooted wednesday afternoon

Wed, 14 Dec 2016 13:39:00 GMT

Maintenance window for Bianca Wednesday 2016-12-12 -- FINISHED

Mon, 12 Dec 2016 12:04:00 GMT

Issues with /gulo during the week end.

Mon, 12 Dec 2016 08:14:00 GMT

On saturday afternoon (roughly 12-14), some parts of the storage system mounted under /gulo (used for glob until it's decomissioned in January) was unavailable. Some jobs may have timed out because of this.

Maintenance window Wednesday 2016-12-07 -- FINISHED

Wed, 07 Dec 2016 08:39:00 GMT

Pica under heavy load

Mon, 05 Dec 2016 14:19:00 GMT

Pica has been under heavy load since this morning.

Login server Milou1 rebooted on Thursday morning

Thu, 24 Nov 2016 08:22:00 GMT

Login server Milou1 rebooted on Friday afternoon

Mon, 21 Nov 2016 09:52:00 GMT

Decommissioning of Gulo on 2017-01-18

Fri, 18 Nov 2016 13:27:00 GMT

Login server Tintin1 rebooted on November 17 19:05

Fri, 18 Nov 2016 08:28:00 GMT

Login server Tintin2 rebooted at 18:15 on November 15

Wed, 16 Nov 2016 12:55:00 GMT

Login server Tintin2 rebooted at 18:15 on November 15

Maintenance on storage system Lupus 2016-11-15 -- now CLOSED

Mon, 14 Nov 2016 12:02:00 GMT

UPPMAX's webexport service is down until next week

Fri, 04 Nov 2016 16:28:00 GMT

Problem changing password - resolved

Thu, 03 Nov 2016 10:53:00 GMT

Hi all,

Our directory service seems to have revoked the right to change password. If you try using the 'passwd' command you will receive a lengthy INSUFFICIENT_ACCESS_RIGHTS error message. This problem is being investigated.

Update 20161107: This has been resolved during the week end.

Maintenance window Wednesday 2016-11-02 -- NOW FINISHED

Wed, 02 Nov 2016 07:40:00 GMT

Tintin network problem

Tue, 01 Nov 2016 07:23:00 GMT

Yesterday from about 13:00 to 17:00 we had network problems within the Tintin cluster that resulted in 122 nodes going offline and jobs crashing.

Issues with Mosler - resolved

Sat, 29 Oct 2016 15:48:00 GMT

Mosler/topolino had issued since the evening of Fri Oct 28 (logins may have faled and jobs may have crashed). These issues should be resolved now (Sat Oct 29).

RESOLVED Issue: Password prompt appears when running 'interactive' jobs

Thu, 27 Oct 2016 13:17:00 GMT

Security update on Milou, Fysast1 and Tintin -- NOW FINISHED

Wed, 26 Oct 2016 13:45:00 GMT

Irma will be restarted to complete security updates -- NOW FINISHED

Tue, 25 Oct 2016 10:27:00 GMT

Many Fysast1 and Milou nodes crashed on Saturday morning (2016-10-22) -- NOW BACK IN PRODUCTION

Sun, 23 Oct 2016 09:32:00 GMT

Problems with login node Milou1 and storage system Gulo -- NOW SOLVED

Wed, 19 Oct 2016 09:33:00 GMT

Problems with Irma's storage system Lupus -- PREPARING FOR A REPAIR

Wed, 19 Oct 2016 09:01:00 GMT

Lower support level during Tuesday 2016-10-18

Mon, 17 Oct 2016 11:12:00 GMT

Maintenance window for Smog Wednesday 2016-10-19 -- NOW FINISHED

Mon, 17 Oct 2016 10:26:00 GMT

Problems with project storage (pica file system v8)

Sun, 16 Oct 2016 20:33:00 GMT

During the week end we had issues with the project storage (file system v8 on pica) which was inaccessible for some time.

Jobs trying to access that part of the storage during this time may have crashed.

Topolino/Mosler available for all users again

Thu, 06 Oct 2016 13:10:00 GMT

Problems with Pica during first weekend of October

Sun, 02 Oct 2016 09:21:00 GMT

Maintenance window Wednesday 2016-10-05 -- NOW FINISHED

Fri, 30 Sep 2016 12:17:00 GMT

Problems with Pica

Thu, 29 Sep 2016 11:59:00 GMT

Problem with /pica/w1, part of storage system Pica

Sun, 25 Sep 2016 11:25:00 GMT

Some Milou and Tintin nodes crashed

Wed, 21 Sep 2016 11:15:00 GMT

Problems yesterday 2016-09-18 to connect to UPPMAX

Mon, 19 Sep 2016 14:57:00 GMT

Problem with a part of Pica, /pica/v19. -- NOW FIXED

Thu, 15 Sep 2016 14:00:00 GMT

Problems with storage system Gulo -- NOW HOPEFULLY FIXED

Fri, 09 Sep 2016 13:41:00 GMT

Maintenance window Wednesday 2016-09-07 -- NOW FINISHED

Wed, 31 Aug 2016 14:05:00 GMT

Reboot of Milou1 due to slow interactive responses

Fri, 26 Aug 2016 15:55:00 GMT

Several jobs lost on Tintin early Sunday morning, 2016-08-14

Wed, 17 Aug 2016 07:27:00 GMT

No maintenance planned for maintenance window 2016-08-03

Thu, 28 Jul 2016 09:43:00 GMT

We lost a small part of our user database (LDAP)

Wed, 20 Jul 2016 13:17:00 GMT

Most of the Milou nodes went down on Thursday evening

Fri, 15 Jul 2016 08:40:00 GMT

Login server Milou2 rebooted on Friday afternoon

Mon, 04 Jul 2016 10:00:00 GMT

Problems with volumes v1 and v10 of storage system Pica -- NOW FIXED

Fri, 01 Jul 2016 06:50:00 GMT

Maintenance window Wednesday 2016-07-06 -- CLOSED SINCE LATE WEDNESDAY AFTERNOON

Fri, 01 Jul 2016 05:55:00 GMT

Login server Milou1 rebooted very early Sunday morning 2016-06-26

Mon, 27 Jun 2016 07:37:00 GMT

Milou1 rebooted thursday morning 3am

Thu, 23 Jun 2016 07:30:00 GMT

Milou1 rebooted for unknown reasons.

2016-06-01 - Maintenance window Wednesday -- NOW CLOSED

Mon, 30 May 2016 11:12:00 GMT

Monthly maintenance window begins at 0900 hours on the first of June.

This time we will:

Upgrade kernel and other system software on all nodes of Irma
Upgrade system software of Irma's storage system Lupus
Upgrade system software on Mosler
Upgrade system software on Smog

We will stop Slurm queues on Irma and Mosler, but not on our other systems.

There is a substantial risk that the maintenance of Mosler and Smog will not finish within one day.

To upgrade system software on Lupus, we need a confirmation from the vendor, that we have not yet received . If we do not get it before Tuesday, we will probably cancel the upgrade and re-open the Slurm queue on Irma on Monday or Tuesday.

We plan no maintenance on our storage systems and on our Linux clusters Fysast1, Milou, and Tintin.

During the maintenance day(s), we will keep you informed about our progress by updates on this page.

2016-05-18 - Last maintenance window for Nestor on Wednesday -- NOW CLOSED

Mon, 16 May 2016 10:59:00 GMT

UPPMAX support for cluster Nestor and storage system Apus ends in May, i.e. this month.

Because of that, we have created a maintenance reservation in Slurm, starting at nine in the morning. After that time, Nestor can be used no more.

You need to evacuate the system before the 18th of May, i.e. copy all important files to your department or to other appropriate storage, because UPPMAX plans to erase all data on Nestor and possibly all data on Apus (i.e. /home directories, /glob directories, /proj directories, /proj/nobackup directories, and other directories).