ICX Stack upgrade problem

  • 1
  • Question
  • Updated 2 months ago
Working on what should be a very straight forward task but becoming a nightmare.  We have some ICX-7450 stacks (a 2x and 5x member stack).  The task is to upgrade from 08.0.30t to 08.0.90a.   Thought was to tftp the 08.0.90a code to the secondary along with tftp'ing over the new 10.1.15T boot loader.  

The TFTP's complete successfully and a sh flash shows what one would think is a working configuration.  Reboot the stack with the 'boot system flash secondary yes' command.  Both units in the stack come up successfully.  Here is where the frustration starts.  Unit 2 reboots due to a stack election (not totally sure why the election occurs when the stack is already been working).  Upon this second reboot the boot load reverts back to 10.1.06T and then gives a invalid bootm command and won't load the 08.0.90a code.  

I have tried to update the uboot with the bootm functionality. Works once then reverts back again.  

So I end up rolling the secondary back to 08.0.30t to get the stack stable again.  I then try the same process but using the primary. Again upgrade seems to work but ultimately a unit in the stack reboot and reverts back to older bootloader and has bootm & kernel errors.  

I can upgrade stand-alones all day long no issues,  but stacks just aren't liking the 08.0.90a upgrade for some reason.  

Anyone else seen this or have some suggestion to get past this?  

Thank you!
Photo of tgash70

tgash70

  • 2 Posts
  • 0 Reply Likes

Posted 2 months ago

  • 1
Photo of Sal Buscemi

Sal Buscemi

  • 1 Post
  • 0 Reply Likes
Check the upgrade notes. I believe you need two steps to go from 8030 to 8090. We do an upgrade to 8080 first, then the newer UFI 8090 
Photo of NETWizz

NETWizz

  • 182 Posts
  • 58 Reply Likes
I have not run into this issue, but that is a HUGE upgrade from 10.1.06T to 10.1.15T and from 08.0.30t to 08.0.90a.  I generally always upgrade code before deploying new devices and at regular intervals, and I am certain more testing has taken place between each build family and the next build family of code.

Most of those didn't even ship from the factory with 08.0.30 builds but rather 08.0.60 originally, which doesn't support stacking... until 08.0.61 builds.

I would make sure you are in configure terminal when you tell it which flash image to boot.  Then do a "write mem" before reloading.

Did you validate that it copied the image to all devices by doing a show flash?

Personally, I would probably jump build families in stages going to say 08.0.61x... Being sure everything works and saving the configuration again.

Then I would then jump to the 08.0.70 builds.  Each family runs a pre-parse script and converts and changes the configuration a little.  I am not 100% sure that 08.0.90 builds can pre-parse 08.0.30 configurations being how different they are for things like LAGs, Dual-Mode, and PoE

***

Honestly, it might be easier to export your configuration and compare it against your template to make everything you need for a clean reconfiguration.

Maybe someone from Ruckus can chime in on this.


Photo of Jon Maiman

Jon Maiman, Employee

  • 11 Posts
  • 5 Reply Likes
Starting with 8.0.90 we introduced the UFI (Unified Fastiron Image).   When upgrading from pre 8.0.90 code, you have to first upgrade to 8.0.90 and then do a second upgrade to 8.0.90ufi.   If you skip the intermediate steps, it will look like the switch is upgraded but some features will not work correctly.   The release notes and the upgrade guide cover the process in detail.

--Jon

Photo of NETWizz

NETWizz

  • 181 Posts
  • 57 Reply Likes

Honestly, the release notes are not very clear on this.  All I could find is:
"Any systems upgraded from 08.0.70 or earlier releases directly to 08.0.90 manually or using the manifest file must be upgraded a second time using the UFI image. If the upgrade is from 08.0.80, then use the UFI image"

Does the Manifest automatically use the UFI image?  Clearly a "show version" will indicate if it installed the UFI build or not.

That said...


Please confirm this works perfectly fine from 08.0.80 builds to install 08.0.90c UFI by doing the Manifest installs such as:

copy tftp system-manifest 10.1.2.3 FI08090c_Manifest.txt primary switch-image
copy tftp system-manifest 10.1.2.3 FI08090c_Manifest.txt secondary router-image


OR something like

copy tftp flash 10.1.2.3 SPS08090cufi.bin primary



Which is it?  Does it likely even matter?

Thank you
(Edited)
Photo of tgash70

tgash70

  • 2 Posts
  • 0 Reply Likes

Thank you everyone for you replies.  Yes I understand this is a HUGE jump.  Problem is this is a hospital environment.  There was ZERO maintenance windows for very long time.  Finally getting the approval patch/upgrade devices.  

Yes,  I have read a number of release notes and upgrade guides,  watched Youtube videos (Ruckus Education and Terry Henry video),  and tested in our lab with spares.  

NetWizz,  yes I have validated images with show commands, etc.  

I will go back to lab with the suggestions of doing a multiple step upgrade from family to family. 

Thanks for the input. 


Photo of NETWizz

NETWizz

  • 171 Posts
  • 45 Reply Likes
I am in the same boat with regard to maintenance windows being State Government for a statewide agency, but thankfully, I doubt it would be hard for me to upgrade from 08.0.80e to 08.0.90c, which is my next upgrade project.

For your project, I would open a TAC case to get a second opinion as well as validate I have backups AND make a reconfiguration template ahead of time in case things go very bad.  Then I would get the same number of switches and build it in a lab on my work bench applying the same firmware/boot rom, same stacking cable connections, and configuration.  Then I would test the upgrade.

Most likely you cannot do a manifest upgrade nor a UFI upgrade, so you would just send over the boot rom and the firmware; I think..  Then when it completes upgrade again to get UFI support up to and including Smart Zone, but this is where I would get a TAC case going.  What I don't know is what happens if 08.0.90c sees dual-mode, which is deprecated and LAGs in their old format...  Does it recognize these and or pre-parse these? or does part of the configuration break?

There have been a ton of changes with these being the major ones I remember:

Access lists now have sequence numbers
Inline Power is on by default for PoE+
There are now lag virtual interfaces
dual-mode is now just an tagged interface under a VLAN which happens to be tagged in other vlan(s)
Oh, and there is a default username/password, which is a non-issue if you already have that configured.

Photo of Ben

Ben, Employee

  • 80 Posts
  • 26 Reply Likes
Hey all, 

I believe this is an issue that will only occur on stacks. It has to do with us moving from one to two boot codes along with some system max changes that have been made from release to release. The standby unit falls into a boot code mismatch when making this large jump. Please use the following upgrade path as it has been deemed safe:

8030 > 8080e (copy boot/OS images separately via TFTP) > 8090 (use UFI image)

If you are making a large code jump, use 8080e legacy images (non-UFI) as the intermediate stop and you should be okay. You can then upgrade to 8090+ via UFI image.