System News

GPUs now available in Bianca

Thu, 09 Sep 2021 12:12:00 GMT

UPPMAX is happy to announce that Bianca now provides GPUs available for all its users! GPUs are a type of accelerator that may significantly increase performance for certain workloads, for instance, workloads that contain large amounts of identical operations on independent pieces of data (so called data parallellism).

Issue with 'interactive' and creating slurm.out files

Tue, 27 Mar 2018 14:07:00 GMT

Problem with Slurm on Rackham and Milou

Mon, 26 Mar 2018 09:46:00 GMT

There is currently a problem with the Slurm master node which affects users on Rackham and Milou. We are investigating.

Configuration problems on Milou and Irma - SOLVED

Thu, 22 Mar 2018 16:16:00 GMT

March maintenance day -- UPDATED Thursday 07:00

Wed, 07 Mar 2018 06:27:00 GMT

Wednesday 7th of March UPPMAX began our monthly service window. Systems and services may become unreachable during the day.

Slow home directories

Wed, 28 Feb 2018 15:25:00 GMT

Home directories have occasionally been extremely slow today. Nothing seems broken but the system is under a lot of pressure from time to time.

Rackham login issues -- SOLVED

Mon, 26 Feb 2018 07:40:00 GMT

We are currently seeing and receiving reports on login issues on Rackham.

Files and directories may be hidden on Bianca -- SOLVED Wednesday

Wed, 21 Feb 2018 13:43:00 GMT

We have received reports of missing files and directories inside the /proj and /proj/nobackup directories on Bianca. Upon inspection the files are actually there, but are not shown by the "ls" command. If you are working on Bianca, you should be aware of this as for example jobs of type “process all files in directory X and compile the result” might finish fine but create false results due to missing input, thus risking incorrect results and conclusions.

A workaround was implemented on Wednesday 2018-03-08 that mitigates this issue.

The fat (256GB) Rackham nodes is currently unavailable -- SOLVED

Mon, 19 Feb 2018 16:15:00 GMT

The fat (256GB) Rackham nodes is currently unavailable due to an issue with Slurm. We are investigating this issue.

Rackham's storage system -- MONDAY: Queues released

Mon, 12 Feb 2018 14:16:00 GMT

Due to an issue with the storage system Crex the Slurm queue on Rackham is currently on hold. This is a summary of the problem.

No new jobs on Rackham 2018-02-09 11:15

Fri, 09 Feb 2018 10:16:00 GMT

We are experiencing problems with crex, the file system on Rackham. In order to not put more strain on the filesystems we will not allow new jobs to start at the moment. If you submit jobs they will be held in the queue.

Fysast1 online

Fri, 09 Feb 2018 08:39:00 GMT

The fysast1 cluster is back online

Milou online

Fri, 09 Feb 2018 08:24:00 GMT

The Milou cluster is back on line

Rackham online

Fri, 09 Feb 2018 07:54:00 GMT

The Rackham cluster is back online

Bianca online again

Thu, 08 Feb 2018 15:10:00 GMT

The Bianca cluster is back online following our service window.

Maintenance window Wednesday 2018-02-07 -- CLOSED

Tue, 06 Feb 2018 12:39:00 GMT

For the February service we will install our new UPS, update Slurm on all clusters, extend the capacity of the storage system Lupus (for Irma), and of course perform the standard kernel and security updates.

The UPPMAX Cloud region will be unavailable Thursday 17:00-20:00 CET

Wed, 31 Jan 2018 13:37:00 GMT

A central switch will be restarted tomorrow Thursday 2018-01-31. The cloud will become temporarily unavailable from the outside i.e. Internet.

Problems with the 'interactive' and Slurm commands on Rackham

Mon, 29 Jan 2018 14:16:00 GMT

The Slurm master on Rackham is currently overloaded and you may experience sluggish Slurm behavior or timeout issues when running commands such as interactive, jobinfo and squeue. We are investigating this issue.

Some projects volumes on pica are slow

Wed, 24 Jan 2018 08:24:00 GMT

Some projects volumes on pica are slow, this may also possibly affect home directories.

Login issue for new Bianca projects -- FIXED

Tue, 16 Jan 2018 08:23:00 GMT

A network problem has been detected on Bianca causing logins to fail for a few of the most recent Bianca projects . We are working on fixing the problem, and expect to Bianca fully working again very soon.

Maintenance window -- COMPLETED

Wed, 10 Jan 2018 08:41:00 GMT

Wednesday 2018-01-10

Monthly maintenance window begins at 0900 hours on the first Wednesday of the month. (That is today.)

This time we will:

Upgrade Slurm, Linux kernel and other system software on Bianca, Dis, Fysast1, Irma, Milou, and Rackham.
Upgrade Linux kernel and other system software on Castor and Grus.

Bianca and Grus will be unavailable while we service them.

We will restart all login nodes of Fysast1, Irma, Milou and Rackham, probably only once.

Slurm queues on Fysast1, Irma, Milou and Rackham will be stopped, but access to Slurm commands will mostly work during the day.

Slurm queues on Bianca will be stopped and, most of the day, logins to Bianca will not be possible.

We plan to keep you informed about out progress with the maintenance with updates here.

UPDATE 2018-01-10, 16:00

Irma is up and running. Bianca, Milou, Rackham and Fysast1 are still down. We will continue security upgrades tomorrow (Thursday) morning.

UPDATE 2018-01-11, 15:15

Irma, Milou, Rackham and Fysast1 are up and running. Bianca is still being tested. Hopefully Bianca will be back today.

UPDATE 2018-01-11, 16:00

Bianca is now up, however, graphical login is not working right now. Text login works fine (http://www.uppmax.uu.se/support/user-guides/bianca-user-guide/).

We're still working on Dis and expect it to be up by tomorrow.

Extension of lupus

Mon, 08 Jan 2018 14:24:00 GMT

The vendor visited us last week and did the physical installation of the lupus extension. Unfortunately, some parts were not correct and we're currently waiting for exchanges that are expected to arrive this week.

UPPMAX staff back after the holidays

Mon, 08 Jan 2018 10:14:00 GMT

We hope 2018 has been good to you so far! UPPMAX staff is back after the holidays and we're focusing on support tickets that have built up over the holidays.

Reduced staff availability over the coming holidays combined with lots of tickets

Mon, 18 Dec 2017 08:54:00 GMT

Most of our staff is on vacation over the coming holidays. You can contact us using regular channels, but response times for support questions might be longer than normal. We are sorry for the inconvenience.

First week of January, most of us are back again.

We also have a lot of tickets about transfer from Milou to Rackham/Bianca and we think there might be hundreds of last minute requests in January. Be prepared the process of getting a transfer project takes some time.

If you want to continue your Milou project, make sure you have applied for a storage project and compute project on Rackham (for non-sensitive data), or a project on Bianca (for sensitive data). http://uppmax.uu.se/support/getting-started/moving-your-research-from-milou-to-rackham/

Creation of new Bianca projects currently on hold -- FIXED

Thu, 14 Dec 2017 12:57:00 GMT

The creation of new Bianca project are currently on hold. If your project is scheduled to start today you will be unable to login.

milou2 rebooted on Friday 2017-12-08 at 03:52

Fri, 08 Dec 2017 07:35:00 GMT

milou2 rebooted on Friday 2017-12-08 at 03:52

milou2 rebooted on Wednesday 2017-12-06 at 03:58

Wed, 06 Dec 2017 07:30:00 GMT

milou2 rebooted on Wednesday 2017-12-06 at 03:58

Updates from SUPR are temporarily disabled

Mon, 04 Dec 2017 12:42:00 GMT

We are performing a change in our infrastructure today starting at 13:00. This change will temporarily stop updates from SUPR reaching UPPMAX. If you have for example recently joined or added a member to a project, you will have to wait before the change becomes visible at UPPMAX.

Fix for broken SSH-connections to the UPPMAX Cloud

Mon, 04 Dec 2017 09:42:00 GMT

If you regularly end up with broken SSH-connections ("broken pipe") to your virtual machine in the UPPMAX region, please use the SSH option ServerAliveInterval. See below for an example.

Issue with the Intel License server

Mon, 27 Nov 2017 07:09:00 GMT

At this moment there is an issue with the Intel license server. You will be unable to use the icc compiler and Intel tools until this issue is resolved.

UPPMAX support low on staff Monday 20/11

Fri, 17 Nov 2017 15:36:00 GMT

The UPPMAX support will be low on staff on Monday 2017-11-20 due to conference.

How to get a high job priority on Bianca

Thu, 16 Nov 2017 15:31:00 GMT

Support ticket system temporarily down --FIXED

Wed, 15 Nov 2017 12:28:00 GMT

Our support email address support@uppmax.uu.se was down for a couple of hours, but is back in service again.

Logging in to Bianca without Rackham

Fri, 10 Nov 2017 09:54:00 GMT

Bianca users outside of SUNET will be unable to login using rackham.uppmax.uu.se. We have created a temporary workaround.

Rackham unavailable -- SOLVED: Rackham available

Thu, 09 Nov 2017 07:59:00 GMT

2017-11-17 09:35 Rackham is now back in regular service.

It was decided to temporarily close down the Rackham cluster last Thursday when several disks on Crex reported themselves broken. The problems now seems solved, and we're awaiting results from the last tests before Rackham is fully back in service.

UPPMAX power outage -- FIXED

Tue, 07 Nov 2017 14:23:00 GMT

UPPMAX experienced a power outage in the server hall on Tuesday.

Problems with /sw on Bianca (now fixed)

Fri, 03 Nov 2017 08:27:00 GMT

The /sw part of Bianca was lost around 07:30 this morning due to an issue with the storage system. This may have caused failed jobs. The system was fixed 08:40.

Quick upgrade of Slurm 2017-11-02 -- COMPLETED

Thu, 02 Nov 2017 15:06:00 GMT

Maintenance window Wednesday 2017-11-01 -- COMPLETED

Wed, 01 Nov 2017 06:59:00 GMT

Monthly maintenance window begins at 09:00 hours on the first Wednesday of the month. (That is today.)

Issues with /sw/data during the week end

Mon, 23 Oct 2017 10:16:00 GMT

/sw/data from pica may have been unavailable for some jobs during the week end and some jobs may have failed because of this.

UPPMAX support system is down -- SOLVED

Fri, 20 Oct 2017 08:12:00 GMT

RT, the support system UPPMAX and all the rest of SNIC is using, is down.
It is located at NSC at Linköping University and the whole university has network problems.

This will make all email to and from support@uppmax.uu.se delayed until the network problem is fixed. So answers to Your support tickets will be delayed.

UPDATE: 11:45

We now have contact with our support system and emails to support@uppmax.uu.se are reaching us again.

Slow home direcotories

Fri, 06 Oct 2017 12:13:00 GMT

Someone seems to be running something very I/O-heavy from the home directories. We are looking for these jobs and will terminate them if found, but it's less than certain that we'll find them.

UPDATE: 14:45
We found the guilty jobs and are termintating them and have notified the user not to do that again.

Accident on Irma caused jobs to fail with status NODE_FAIL

Wed, 04 Oct 2017 15:42:00 GMT

We sadly inform you that today at 17:02:37 a human error caused the compute nodes on Irma to reboot. The jobs running was canceled and will show up with status NODE_FAIL. The accident occured while investigating an issue with the storage network. We are very sorry about this.

UPPMAX shutdown due to cooling failure -- FIXED

Tue, 26 Sep 2017 18:32:00 GMT

lupus failover issue -- FIXED

Mon, 25 Sep 2017 10:44:00 GMT

Maintenance indication in output from command jobinfo

Mon, 25 Sep 2017 10:35:00 GMT

UPPMAX made a small change in "jobinfo" output.

In the REASON column for waiting jobs, "(Maintenance)" is shown for jobs that can not start before the next maintenance reservation.

Please note that maintenance reservations many times are moved forward to next month before the actual maintenance window.

Many Irma compute nodes lost electric power -- FIXED

Mon, 25 Sep 2017 07:05:00 GMT

Three racks of Irma's compute nodes lost power,because an automatic fuse shut down.

Some jobs were lost due to this. We are very sorry about that. Please rerun those jobs that were affected.

It looks like nodes i[167-250] were affected.

So what was the reason? It looks like an ethernet switch diied, possibly short circuited, so the automatic fuse shut down, getting more switches and the compute nodes to go down.

We have error reported to our support vendor. Until the bad ethernet switch has been repaired or replaced, Irma runs with a fewer number of compute nodes.

Update at 0950 hours

Now only nodes i[179-226] are down.

Maintenance window Wednesday 2017-09-06" -- FINISHED

Wed, 06 Sep 2017 06:09:00 GMT

milou2 rebooted August 28

Tue, 29 Aug 2017 07:22:00 GMT

milou2 rebooted Monday 2017-08-28 at 19:51.

Replacing (nearly) all disks on Irma's compute nodes -- DONE

Wed, 23 Aug 2017 05:44:00 GMT

Restarting irma-q

Tue, 22 Aug 2017 06:12:00 GMT

We're restarting irma-q for technical reasons. The slurm queue system may be unavailable for submitting/verifying job status for a few minutes.

milou2 rebooted August 19

Mon, 21 Aug 2017 07:13:00 GMT

milou2 rebooted on Saturday 2017-08-19.

Bianca's storage system Castor had a hiccup yesterday Thursday -- FIXED

Fri, 04 Aug 2017 08:14:00 GMT

Maintenance window Wednesday 2017-08-02 -- FINISHED

Wed, 02 Aug 2017 06:37:00 GMT

Unexpected reboot of Pica at Monday morning.

Mon, 24 Jul 2017 08:06:00 GMT

Restart of two Milou login servers today Thursday

Thu, 20 Jul 2017 14:01:00 GMT

Lower service level during UPPMAX holidays

Thu, 13 Jul 2017 08:54:00 GMT

Part of storage system Pica is still very slow

Wed, 12 Jul 2017 18:00:00 GMT

Pica was partly restarted just now, please look for problems in your job output

Tue, 11 Jul 2017 11:14:00 GMT

UPPMAX had to restart part of storage system Pica, because it worked too slowly with nearly no read/write traffic.

The restart was done a little after 1300 hours.

For Rackham users, this meant that you might have had problems with reading and writing to your home directory.

For Milou users, this meant that you also might have had problems with reading and writing to your home directory. But for Milou users, also reading from /sw (where the modules live) and reading and writing to some project directories were affected.

Please look one extra time for problems in your job output, for jobs running at this time.

We are sorry for the inconvenience.

On Milou and Rackham, very difficult to login or otherwise use /home directories -- FIXED

Tue, 11 Jul 2017 09:26:00 GMT

UPPMAX has problem with an extremely slow access to /sw (where e.g. modules live) and home directories on Milou, and to home directories on Rackham.

Because of that, it is very difficult to login to Milou and Rackham.

We will investigate the source of this problem, and will report any success as updates here.

Update at 1310 hours

We restarted part of Pica, and that solved the problem

Hopefully your jobs will continue without problems, but please be careful and look once extra time for errors in your job output.

SUPR and C3SE website down

Tue, 11 Jul 2017 06:47:00 GMT

SUPR and C3SE websites are down at the moment. This prevents you from using SUPR at the moment. Please try again later

No maintenance planned for today's maintenance window

Wed, 05 Jul 2017 06:57:00 GMT

First (non-holiday) Wednesday of each month is UPPMAX's normal, planned maintenance window.

But today we will do no maintenance.

Happy computing!

Next maintenance window is 2nd of August.

Restart of login server milou-f Tuesday morning -- FINISHED

Tue, 04 Jul 2017 08:55:00 GMT

File system mounts of Pica volumes was not working correctly.

This was fixed by a restart of the server. Now it works much better.

We are sorry about any inconvenience for you due to this.

Lost contact with Milou nodes m[1-48] for an hour this morning -- FIXED

Mon, 03 Jul 2017 07:35:00 GMT

From approximately 0800 hours to 0910 hours this morning, an ethernet switch in Milou lost power, making 48 nodes unavailable.

Two jobs got NODE_FAIL when trying to start, and interactive work on these nodes was denied. Otherwise, we seem to have had no problems with the temporary network loss.

Singularity is available

Tue, 27 Jun 2017 06:11:00 GMT

The container engine Singularity is available on UPPMAX systems now. See the Singularity users guide for more information.

Urgent kernel upgrade -- FINISHED

Wed, 21 Jun 2017 07:01:00 GMT

Today we are performing an urgent kernel upgrade on Milou, Fysast1, Rackham, Irma, and Bianca. Login nodes will be restarted during the day. No running jorbs or queues are stopped. We will update on the progress here in System News during the day.

UPDATE 16:00 - Update completed.

Bianca graphical login now working

Mon, 12 Jun 2017 09:33:00 GMT

Uses Thinlinc Web Access. Not X-forwardning.

Bianca's storage system Castor has problems -- FIXED

Sun, 11 Jun 2017 17:47:00 GMT

Maintenance window Wednesday 2017-06-07 -- FINISHED

Wed, 07 Jun 2017 06:06:00 GMT