Have you set the macros STDIO_DEBUG_SERIAL and STDIO_DEBUG_BAUD in your project/program? If so, all debug output goes over that serial port at that baud rate.
Or maybe you're seeing the SPI traffic on serial port B where it's writing the new firmware out to the serial boot flash? I think it shares that port with external pins, and that could be what you're seeing.
It's possible that you're seeing 2 minutes of writing the 400KB to flash, and then another minute as it reads that data back to verify the checksum. If you enable verbose debugging output, that would slow down the process.
Are those times you're seeing with the standard Samples/RemoteProgramUpdate/download_firmware.c sample, or with the update functionality included in your code?