Novatel Ovation MC950D Fails/Hungs Intermittently

[GeorgeVita]

EDIT: tech info added

Mar 29 13:02:59 300PL kernel: [142120.140359] usb 1-2.4: USB disconnect, address 23
Mar 29 13:02:59 300PL kernel: [142120.143494] option 1-2.4:1.0: device disconnected
Mar 29 13:02:59 300PL kernel: [142120.145360] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
Mar 29 13:02:59 300PL kernel: [142120.155013] option 1-2.4:1.1: device disconnected
Mar 29 13:02:59 300PL pppd[15345]: Modem hangup
Mar 29 13:02:59 300PL pppd[15345]: Connect time 307.2 minutes.
Mar 29 13:02:59 300PL pppd[15345]: Sent 2831521 bytes, received 16614180 bytes.
Mar 29 13:02:59 300PL pppd[15345]: Connection terminated.
Mar 29 13:02:59 300PL pppd[15345]: Exit.
Mar 29 13:02:59 300PL kernel: [142120.561284] usb 1-2.4: new full speed USB device using uhci_hcd and address 24
Mar 29 13:02:59 300PL kernel: [142120.601593] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
Mar 29 13:02:59 300PL kernel: [142120.691916] usb 1-2.4: configuration #1 chosen from 1 choice

The above seems to be a h/w reset caused by the modem (circuit or firmware) or a momentary power loss. The modem did not stuck (brown out) but restarted and recognized by the system. Also the system rearranges /dev/ttyUSBx starting from 0 because wvdial exited normally. In another case the connecting program stuck holding the /dev/ttyUSB0 resulting in a rearrangement to /dev/ttyUSB1 & 2

If the above caused by a power problem could be fixed with the patched cable.

--> Disconnecting at Mon Mar 30 11:35:11 2009
--> The PPP daemon has died: A modem hung up the phone (exit code = 16)
--> man pppd explains pppd error codes in more detail.
--> Try again and look into /var/log/messages and the wvdial and pppd man pages for more information.
--> Auto Reconnect will be attempted in 5 seconds
--> Cannot open /dev/ttyUSB0: No such file or directory
--> Cannot open /dev/ttyUSB0: No such file or directory
--> Cannot open /dev/ttyUSB0: No such file or directory
--> Disconnecting at Mon Mar 30 11:35:11 2009
...
...If I run wvdial while the modem is in its 'madly blinking' state, I get back

--> Cannot open /dev/ttuUSB0: Device or resource busy.

I tried running minicom against the port - it hung 'Initializing Modem'. Killing minicom caused it to hang 'Resetting Modem'. Rebooting Ubuntu did not help because power to the USB port never dropped.

In the above case the modem is in an ERROR state with its firmware knowing or not this Error condition (the 'madly blinking' could be random or a status). We are sure that power removal is a solution.

From your understanding of USB cables, is that all I need to do, or do I need to open/close all four leads?

The best is to open/close at least 3 leads (V+ and data). As it is simpler to open/close only the V+ (red cable) you have to try it. If the modem is well designed will reset fully even if the line driver has data signals.

Finally Init2 = AT&FE0V1X1&D2&C1S0=0 it is very good because of AT&F which restores factory defaults, useful when using the modem with windows where we don't know the exact settings (via control panel you can enable and check the modem log file to see some of them).

Using minicom you can try AT&F then ATZ and AT&V to see factory default settings.
(minicom -s starts minicom without initializing showing the main menu)

Regards,
George

[nhoeller]

Latest news... I built a new PC with Windows XP, NAT32 and the official MobiLink software. It ran right through the night, I tried to get technical support at Novatel to call me back. Shortly after hanging up the phone, the Novatel modem hung - same symptoms. MobiLink was unable to connect to the device.

Since I have the problem on different hardware and different software (with and without the 'USB cable patch' to rule out stray electrical gremlins), I have convinced the wireless carrier to replace the modem. Unfortunately, they are only shipping a ZTE modem which appears not to be popular with the Linux community. I may yet be stuck with the MC950D and will need to implement automation to power-cycle the modem.

I will admit that I am getting quite attached to having an Internet connection with 150ms latency, compared to 70+0ms latency of my satellite connection (and the random bandwidth degradation from their Fair Access Policy).

[GeorgeVita]

...
Since I have the problem on different hardware and different software...

We must think that it is a modem's fault.

Can you do a final h/w test? Power the modem with an external 5VDC power supply rated at least 1A (best is a linear type with a 7805 regulator good heatsink and capacitors over 1000uF) or even better a bench PSU.

What is the signal strength in your position? (see it from WinXP or with the AT+CSQ from minicom)

Some ZTE types are working like: MF626, MF628, MF632 AND MF627 AND MF636 (mine) with modifications.

If it helps I could do some tests with my MF636 and local provider for comparison.

G

[nhoeller]

George, I currently have the modem plugged into a powered USB hub. The power supply is rated at 2.5A - it is one of the regulated power supplies, but I have no idea what the internals look like.

I was so annoyed I forgot to ask what model of ZTE they were shipping me. I should get it shortly.

Signal strength is typically 3 bars out of 5. Sometimes I see 4 bars, sometimes it drops to 2.

Is there a way to check signal strength while wvdial is using the modem? I trying pointing minicom at /dev/ttyUSB1, but got no answer.

[GeorgeVita]

When I was using a Huawei E170, been connected via /dev/ttyUSB0 could used minicom to test some AT commands via /dev/ttyUSB1. Setup minicom before your efforts or use minicom -s and serial port setup from the menu. In case of misspelling it stucks and usually ... you have to reboot!

I am going to test the above with the ZTE.
EDIT:> ok worked, not fully tested.
Modem attached as /dev/ttyUSB0-1-2-3 connected via /dev/ttyUSB3 minicom at /dev/ttyUSB1
Tested with AT, AT+CSQ and ATI

NEW EDIT: Can you check from: System > Administration > System Log > click on daemon.log
scroll down and find the lines regarding the ttyUSBx (modem) at disconnection time. My idea is a problem caused by a System or modem timeout. System timeout when idle/suspend/screensaver. My modem disappeared from Network Manager after screen saving! I could not connect till I made a real h/w disconnection. Lets try to debug it.

G

[nhoeller]

George, I received a ZTE MF636 as a replacement for the Novatel. After enabling and testing the device on Vista, I moved it to the Ubuntu server. Although usb_modeswitch seemed to work initially, it would not switch the device out of its ZeroCD mode after I rebooted. Your trick of sending an AT+ZCDRUN=8 command to the modem to disable ZeroCD saved the day. In general, the ZTE modem appears to deliver better throughput than the Novatel.

The modem intermittently hangs just like the Novatel, although the symptoms are different. I see the following errors in syslog:

Apr 23 03:51:33 300PL pppd[16581]: No response to 4 echo-requests
Apr 23 03:51:33 300PL pppd[16581]: Serial link appears to be disconnected.
Apr 23 03:51:33 300PL pppd[16581]: Connect time 601.6 minutes.
Apr 23 03:51:33 300PL pppd[16581]: Sent 26748373 bytes, received 132386910 bytes.
Apr 23 03:51:39 300PL pppd[16581]: Connection terminated.

The modem does not disconnect from the USB port - the /dev/ttyUSB0-2 devices still exist. Sometimes the modem shows a solid green light (indicating that it is registered but not connected on a 2G network), sometimes the modem light goes off completely. In the former state, wvdial cannot communicate with the modem (Modem not responding to ATZ commands). daemon.log shows nothing relevant to the problem.

Since I have connection issues with multiple modems on multiple PCs and operating systems, I suspect the modems are not properly handling some sort of carrier problem. Unfortunately, the carrier has not stated any sort of Linux support for the ZTE and reproducing the problem on Windows will have to wait.

I have installed your cable patch to eliminate power issues. The next step is to build a device that will interrupt and restore USB power under control of the Ubuntu server. I am hoping I can also use this device to power off the modem during server bootup - Ubuntu hangs during hardware detection if the modem is plugged in (dmesg shows the modem is detected and /dev/ttyUSB0-2 ports are added but the filesystems have not been mounted). It may be a problem with my udev rules.

[GeorgeVita]

Hi Norbert,
taking account of the differences we have in our s/w and h/w configuration, I can only give some thoughts for your debugging.

...The modem intermittently hangs just like the Novatel, although the symptoms are different. I see the following errors in syslog:

Apr 23 03:51:33 300PL pppd[16581]: No response to 4 echo-requests
Apr 23 03:51:33 300PL pppd[16581]: Serial link appears to be disconnected.
Apr 23 03:51:33 300PL pppd[16581]: Connect time 601.6 minutes.
Apr 23 03:51:33 300PL pppd[16581]: Sent 26748373 bytes, received 132386910 bytes.
Apr 23 03:51:39 300PL pppd[16581]: Connection terminated.

The modem does not disconnect from the USB port - the /dev/ttyUSB0-2 devices still exist.

This disconnection could be done from your provider (mobile phone company). They possibly cannot or do not want a 10 hours connection. Another reason could be a s/w or h/w timeout into the modem (?).

...Sometimes the modem shows a solid green light (indicating that it is registered but not connected on a 2G network), sometimes the modem light goes off completely.

This could be the modem's timeout. This can be tested or altered by loading a "dummy" page which reloads itself after a fixed time period. You can make an .html file (I don't know how) or open a tab with a page which has this function (test this one: http://www.emsc-csem.org/).

Using above trick you have to pay attention to the data traffic instead of paying a lot of money to the provider!

...In the former state, wvdial cannot communicate with the modem (Modem not responding to ATZ commands). daemon.log shows nothing relevant to the problem.

The modem must be in a sleep mode or stucked. I have faced this problem with my ZTE MF636 one or two times in the past but I couldn't repeat it in order to debug it.
Also in my case the cable patch fixes the Huawei's h/w reset problem and not the ZTE one. I realize that ZTE works better (data rate) with a small extention cable to gain RF signal or in direct connection with the USB port (using a notebook where the metal parts are less than with a desktop).

Concluding, if your application must be "bullet proof" you have to check your idea to add extra h/w and simulate the manual restart (& replug) condition.

Regards,
George

[nhoeller]

Two steps forward, one step back. The modem has actually been reasonably reliable recently, or rather wvdial detects that the connection has failed and automatically restarts the connection. The modem may stay up for several hundred minutes and then require a restart after only 4 minutes. Except for today (which has been particularly bad), I have only had to manually unplug/plug the modem a few times a week.

I have built a circuit on top of an Arduino controller (http://arduino.cc) that incorporates George's power filter and a relay cut into the USB power line. I can send serial device commands to the Arduino over its USB port. Unfortunately, the current crop of Arduino boards automatically reset themselves when a serial connection is established, as part of support for automating the process of uploading new code to the Arduino. I am testing out various ways of getting around the resets.

One thing I noticed is that the ZTE modem sometimes connects on /dev/ttyUSB0-3, sometimes only on /dev/ttyUSB0-2. It seems that the highest ttyUSB device is the one I need to use for wvdial.

[radolin]

Hi Norbert,

unfortunately I can't help with the modem reset problem, but only about the Arduino reset one .

You can try a software solution by disabling the HUPCL (hang up on close) flag of the virtual COM port, so DTR is not lowered after a disconnect.

The other 100% sure way is a hardware solution - you just need to remove one of the 100nF capacitors - details here http://www.arduino.cc/playground/Mai...rialConnection

Also if your machine has real COM/LPT port you can just control the relay with it and avoid the whole Arduino thing. Since you already have external 5V supply for the USB hub with one transistor, a couple of resistors and a protection diode you'll be able to switch the relay directly.

Btw I am researching what 3G modem to buy and was looking into the ZTE MF626 but am a bit scared after this stories. It will not be my primary inet conection, so if it works for 1-2 hours will be enough but still ...
Any advice?

Regards,
Rado

[nhoeller]

Rado, I will test the -hupcl flag on stty as soon as I get a chance. I did try a Perl script (from the Arduino forum) to disable DTR and send the required characters to the Arduino. It sort of worked, in so far as the Arduino did not reset itself. However, the Arduino still did not recognize the character. I added a trailing '0A'x and had the Arduino echo back what it saw, but must not have the right 'magic incantation'.

Thanks for the tip on using a COM/LPT1 port directly - definitely cheaper than the setup I have now. The current PC has a COM port. I will be replacing it with a laptop as soon as I can free one up - no COM port but it does have an LPT port.

I am not sure my experiences are any reflection on the ZTE MF626. In general, this modem is giving better bandwidth than the Novatel Ovation MC950D. Being able to turn off the ZeroCD is a nice feature - one less thing to worry about on Linux. I would say that it freezes up less often. What I am seeing are temporary conditions where wvdial detects that the connection is not responding and automatically recovers. Most of the time, I do not even notice.

I do not know if this happened with the Novatel modem, since at the time I was not monitoring the logs as closely as I am doing now. My suspicion is that the problem is a glitch in the carrier's network. It is even possible that the connection would come back by itself if I disabled wvdial recovery. I also suspect that the modem freezes are caused by something happening at the carrier - they seem related to a switch from HSUPA to EDGE.
Regards, Norbert