Posible iSCSI problem

**TheR** · April 22nd, 2011

Ubuntu 10.4 on all servers.

I am using ubuntu for my KVM servers connected with 10g Ethernet to ubuntu based iSCSI server. Everything seems to work OK for almost a year now. The only problem (which seems like it is not critical) is that at the night, when I am doing my LVM snapshots and making copies of virtual machines, once or twice a week I get this error message on one of windows machine system log:

Program: Viostor
Error id: 9
Description: The device, \Device\Scsi\viostor2, did not respond within the timeout period.

It looks like iSCSI server was to busy during snapshot and virtualized server got timeout. HW shouldn't be problems since all servers are using i7 xeons.

Can you suggest iSCSI server or maybe client tweaking to avoid this?

by
TheR

**usatonycuba** · April 22nd, 2011

Originally Posted by TheR

...is that at the night, when I am doing my LVM snapshots and making copies of virtual machines...

Have you change the Default Values in ietd.conf of MaxRecvDataSegmentLenth and/or MaxXmitDataSegmentLenth..?

**TheR** · April 22nd, 2011

All my ietd.conf settings are still commented out. So:

#MaxRecvDataSegmentLength 8192 # Max data per PDU to receive
#MaxXmitDataSegmentLength 8192 # Max data per PDU to transmit

by
TheR

**usatonycuba** · April 22nd, 2011

Originally Posted by TheR

All my ietd.conf settings are still commented out. So:

#MaxRecvDataSegmentLength 8192 # Max data per PDU to receive
#MaxXmitDataSegmentLength 8192 # Max data per PDU to transmit

by
TheR

8192 are the default values.... Edited: why not to move them to a little higher value...?.. (If you really are expecting higher values due to your work-processes)..?

PD: My doubt is due, because this happens exactly when you're doing (LVM snapshots and making copies of virtual machines) .. may be an overload ( not due to your i7 xeons) .. but the parameters of the file ietd.conf on the ability/capacity of Data-Lenth..?

Carefully .. remember that each value change in a node Must Be the Same at Other End

**TheR** · April 22nd, 2011

Originally Posted by usatonycuba

PD: My doubt is due, because this happens exactly when you're doing (LVM snapshots and making copies of virtual machines) .. may be an overload ( not due to your i7 xeons) .. but the parameters of the file ietd.conf on the ability/capacity of Data-Lenth..?

Carefully .. remember that each value change in a node Must Be the Same at Other End

This made me think maybe snapshoting is using block that is too large. I am doing dd with bs=64k, simply because it is faster than default.

I will lower bs to 8k and see if there is any difference.

by
TheR

**TheR** · April 23rd, 2011

Originally Posted by TheR

This made me think maybe snapshoting is using block that is too large. I am doing dd with bs=64k, simply because it is faster than default.

I will lower bs to 8k and see if there is any difference.

Nope. No difference. Still got one error.

by
TheR

**usatonycuba** · April 23rd, 2011

When using open-iscsi you can change few parameters to improve it... between (initiator and target).. as you said .. there is Ubuntu 10.4 on all servers... you can get the the whole parameter list (at the indicator side) typing iscsiadm -m session -P 2 result will give you something like:

Code:

************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 8192
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1

and here its ...outlined how to Sets the maximum data segment length that can be received/sent

Code:

       [MaxRecvDataSegmentLength <value>]
              Optional.  Sets  the  maximum  data  segment  length that can be
              received. The <value> should be set to multiples  of  PAGE_SIZE.
              Currently  the  maximum  supported value is 64 * PAGE_SIZE, e.g.
              262144 if PAGE_SIZE is 4kB. Configuring  too  large  values  may
              lead to problems allocating sufficient memory, which in turn may
              lead to SCSI commands timing out  at  the  initiator  host.  The
              default value is 8192.

       [MaxXmitDataSegmentLength <value>]
              Optional. Sets the maximum data segment length that can be sent.
              The    <value>    actually    used    is    the    minimum    of
              MaxXmitDataSegmentLength    and   the   MaxRecvDataSegmentLength
              announced by  the  initiator.  The  <value>  should  be  set  to
              multiples of PAGE_SIZE. Currently the maximum supported value is
              64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB. Configuring too
              large  values may lead to problems allocating sufficient memory,
              which in turn may lead  to  SCSI  commands  timing  out  at  the
              initiator host. The default value is 8192.

It may help

**TheR** · April 24th, 2011

Originally Posted by usatonycuba

When using open-iscsi you can change few parameters to improve it... between (initiator and target).. as you said .. there is Ubuntu 10.4 on all servers... you can get the the whole parameter list (at the indicator side) typing iscsiadm -m session -P 2 result will give you something like:

Code:

************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 8192
FirstBurstLength: 65536
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: No
MaxOutstandingR2T: 1

and here its ...outlined how to Sets the maximum data segment length that can be received/sent

Code:

       [MaxRecvDataSegmentLength <value>]
              Optional.  Sets  the  maximum  data  segment  length that can be
              received. The <value> should be set to multiples  of  PAGE_SIZE.
              Currently  the  maximum  supported value is 64 * PAGE_SIZE, e.g.
              262144 if PAGE_SIZE is 4kB. Configuring  too  large  values  may
              lead to problems allocating sufficient memory, which in turn may
              lead to SCSI commands timing out  at  the  initiator  host.  The
              default value is 8192.

       [MaxXmitDataSegmentLength <value>]
              Optional. Sets the maximum data segment length that can be sent.
              The    <value>    actually    used    is    the    minimum    of
              MaxXmitDataSegmentLength    and   the   MaxRecvDataSegmentLength
              announced by  the  initiator.  The  <value>  should  be  set  to
              multiples of PAGE_SIZE. Currently the maximum supported value is
              64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB. Configuring too
              large  values may lead to problems allocating sufficient memory,
              which in turn may lead  to  SCSI  commands  timing  out  at  the
              initiator host. The default value is 8192.

It may help

Thank you. So you are suggesting lowering MaxRecvDataSegmentLength parameter.

According to usatonycuba this could be a problem since parameters should be changed both on client and initiator.

It is going to be a bit hard since this is working system with 9 active virtualized servers. I do plan to make one software update on iscsi server soon (uptime is close to 200 days now) but am waiting for new usb keys with cache so updating should be a lot faster.

by
TheR

**usatonycuba** · April 24th, 2011

Originally Posted by TheR

Thank you. So you are suggesting lowering MaxRecvDataSegmentLength parameter.

According to usatonycuba this could be a problem since parameters should be changed both on client and initiator.

It is going to be a bit hard since this is working system with 9 active virtualized servers. I do plan to make one software update on iscsi server soon (uptime is close to 200 days now) but am waiting for new usb keys with cache so updating should be a lot faster.

by
TheR

run that command too, at the target side, take a look what you get

Code:

iscsiadm -m session -P 2

MaxRecvDataSegmentLength will be, the maximum size of data that the initiator can receive in just one iSCSI PDU... So, whatever the initiator is able to receive and handle, it will be also what the target is permitted to send

**TheR** · April 29th, 2011

Originally Posted by TheR

Nope. No difference. Still got one error.

by
TheR

Using bs=4k seems to help. I have had three backups without hickup now.

Backup window has gone from 1.5 hours to 2 hours which is acceptable.

Thank you guys for ideas.

by
TheR