9.10 Upstart Breaks Cron,Cups,SSH, etc...

[cmcginty]

Hi,

Running 9.10 now, but having problems with many of my background services.

* Cups service no longer starts at boot
* SSHd service no longer starts at boot
* Cron jobs are not being run
* Samba file shares and/or printers not initialized correctly at startup.

HELP! What is the deal with Upstart? How can I get it to run my system services again?

Is there a master thread for using Upstart? Does rcconf, invoke-rc.d, and other utilities still work?

------
Update ...

I originally thought this was fixed, but now I see that my services (CUPS, cron) still do not startup on boot. Can anyone confirm this and provide a workaround?

[lswb]

I would like to see some good documentation for upstart too. However, one of the goals of upstart is that services like cups and sshd do not need to start at boot but only when actually needed. So if you were going to print something, cups would start at that time. How they get it to work for printing over a lan I don't know.

I haven't looked at KKs init system but I believe it still has the sysv compatibility layer to run the older style /etc/init.d scripts. Take a look in in some of the /usr/share/doc directories, you may find some info there.

[cmcginty]

I think cron is working alright now. I had to re-install my local crontab file with:


crontab -e

and then making a change and saving it. Still no luck on the other services. If I run:


sudo invoke-rc.d service restart

then the service will start correctly. I'd rather have this run automatically.

[cmcginty]

Should I be worried about this output?


sudo dpkg-reconfigure cups
* Stopping Common Unix Printing System: cupsd [ OK ]
insserv: warning: script 'S07powernowd.early' missing LSB tags and overrides
insserv: warning: script 'atd' missing LSB tags and overrides
insserv: warning: script 'gdm' missing LSB tags and overrides
insserv: warning: script 'hal' missing LSB tags and overrides
insserv: warning: script 'ufw' missing LSB tags and overrides
insserv: warning: script 'dbus' missing LSB tags and overrides
insserv: warning: script 'cron' missing LSB tags and overrides
insserv: warning: current start runlevel(s) (0) of script `halt' overwrites defaults (empty).
insserv: warning: script 'gssd' missing LSB tags and overrides
insserv: warning: script 'udev' missing LSB tags and overrides
insserv: warning: script 'cryptdisks-enable' missing LSB tags and overrides
insserv: warning: current start runlevel(s) (0 6) of script `umountfs' overwrites defaults (empty).
insserv: warning: script 'sreadahead' missing LSB tags and overrides
insserv: warning: script 'acpid' missing LSB tags and overrides
insserv: warning: script 'dmesg' missing LSB tags and overrides
insserv: warning: script 'statd' missing LSB tags and overrides
insserv: warning: script 'rsyslog-kmsg' missing LSB tags and overrides
insserv: warning: script 'network-manager' missing LSB tags and overrides
insserv: warning: script 'udevmonitor' missing LSB tags and overrides
insserv: warning: script 'rsyslog' missing LSB tags and overrides
insserv: warning: script 'avahi-daemon' missing LSB tags and overrides
insserv: warning: script 'module-init-tools' missing LSB tags and overrides
insserv: warning: script 'udevtrigger' missing LSB tags and overrides
insserv: warning: script 'rpc_pipefs' missing LSB tags and overrides
insserv: warning: script 'apport' missing LSB tags and overrides
insserv: warning: current stop runlevel(s) (1) of script `policykit' overwrites defaults (empty).
insserv: warning: current start runlevel(s) (0 6) of script `wpa-ifupdown' overwrites defaults (empty).
insserv: warning: script 'idmapd' missing LSB tags and overrides
insserv: warning: script 'procps' missing LSB tags and overrides
insserv: warning: current start runlevel(s) (6) of script `reboot' overwrites defaults (empty).
insserv: warning: current start runlevel(s) (0 6) of script `sendsigs' overwrites defaults (empty).
insserv: warning: current start runlevel(s) (0 6) of script `umountroot' overwrites defaults (empty).
insserv: warning: script 'usplash' missing LSB tags and overrides
insserv: warning: script 'anacron' missing LSB tags and overrides
insserv: warning: current start runlevel(s) (0 6) of script `networking' overwrites defaults (empty).
insserv: warning: script 'udev-finish' missing LSB tags and overrides
insserv: warning: script 'hwclock' missing LSB tags and overrides
insserv: warning: script 'portmap' missing LSB tags and overrides
insserv: warning: current start runlevel(s) (0 6) of script `umountnfs.sh' overwrites defaults (empty).
insserv: warning: script 'powernowd.early' missing LSB tags and overrides
insserv: There is a loop between service rsyslog and hwclock if stopped
insserv: loop involving service hwclock at depth 4
insserv: loop involving service sysklogd at depth 3
insserv: There is a loop between service rsyslog and sysklogd if started
insserv: loop involving service sysklogd at depth 3
insserv: loop involving service hwclock at depth 2
insserv: loop involving service rsyslog at depth 4
insserv: There is a loop between service hwclock and sysklogd if stopped
insserv: loop involving service rsyslog at depth 5
insserv: exiting without changing boot order!
* Starting Common Unix Printing System: cupsd [ OK ]

[cmcginty]

I figured out the reason for these issue. After the upgrade, the system left the old package configuration files in the /etc/init.d directory. They need to be cleaned out:


aptitude -F %p search "~c" | xargs sudo dpkg -P

After that I reinstalled the packages complaining about loop errors:


sudo apt-get install --reinstall kerneloops kerneloops-daemon rsyslog udev


Then cups can be reinstalled now without errors:


sudo apt-get install --reinstall cups

[PorcelainMouse]

What about SSH? My sshd doesn't start on boot, either, after upgrade to 9.10 Karmic.

[cmcginty]

For the last step do:


sudo apt-get install --reinstall openssh-server

[cmcginty]

bump .. updated question. This bug came back. Has anyone else run into this issue?

[arock99]

ssh, samba, and others are not starting for me as well.

What logs should I be looking at to see clues as to what occured during startup of upstart?

Edit:

I ran "initctl list" and ssh.conf (which i created) did not start.

Here is the content of the file...keep in mind i have never written something for upstart:

# ssh daemon
#

description "ssh daemon"

start on runlevel [2345]
stop on runlevel [!2345]

expect fork
respawn

exec /etc/init.d/ssh start

[arock99]

I "fixed" openssh by doing a complete removal and then installing it again.

...and now I just found out mysql doesnt start automatically anymore since the "upgrade". This "solution" obviously wont work for mysql as i dont want to screw around with the data. With mysql not starting then obviously mediatomb isnt starting automatically either

I really wish i hadnt decided to go to 9.10...its been nothing but trouble since i upgraded earlier this week

Edit:

Its one thing after another...now i just noticed NX isn't starting up automatically anymore...but this one if i try and manually restart it, it wont start

Edit: Now its cvsd i just noticed didnt start up....

almost makes me want to go back to Windows lol

...almost

[bebopfreak]

In my case (I use a labtop with wireless connection only), it broke after the last upgrade.
I found out that there was a tiny difference in the file
'/etc/init/rc-sysinit.conf'
which seems to inhibit sysinit to start when the network isn't ready.
When I used the old version everything worked again.
old start condition:
"start on filesystem"
new start condition:
"start on filesystem and net-device-up IFACE=lo"

[at0m1sk]

upstart 0.6.3-11 was the issue, reverted to 0.6.3-10 and everything is back to normal

this was my issue do not go crazy w/ ur configs try this first

[arock99]

this was my issue do not go crazy w/ ur configs try this first

Thanks...i'll have to look around to see how to install a specific version (unless downloading it from the official site is the only way which is fine)

Edit: I just figured out how to downgrade and rebooted...

everything is working again! Thank You!

[Louigi Verona]

1. Guys, can you please leave instructions for non-tech guys like me how to downgrade?

2. Can you also please look into this thread and see if my problems with start up scripts on light sensor in my laptop can be connected to this?

http://swiss.ubuntuforums.org/showthread.php?t=1361002

[Jon Reck]

On the 20th I innocently upgraded the packages suggested by the Update Manager. Next day upon re-boot I found that Apache and Postgresql no longer started.

Solution: downgrade Upstart 0.6.3-11 to 0.6.3-10. Apache and Postgresql work normally now.

How? Using the Synaptic Package Manager search for upstart. Select it and then under Package click on Force Version. You get to choose between 0.6.3-11 and 0.6.3-10. Choose 10 and click on apply.

it's that easy. But it took me hours of googleing to find the solution. Very frustrating.

- Jon

[Louigi Verona]

Thx, man. I've did this and the system is now back to normal.

Is there anyway to see what has been updated with the recent update to at least know what to downgrade?

[Jon Reck]

/var/log/dpkg.log shows an update history

[diddy1234]

so whenis this going to be fixed with a new version of upstart ?

At the moment I have a completly hosed system.
I downgraded upstart to the previous version and this worked while i had a dynamic address but since my install is a home server, I have to have a fixed ip address and since changing this back to fixed ip address it no longer works.

Id like to re-install the system (as it was an upgrade from 8.04) but since upstart is screwed there is no point installing a new system that will also be non working.

I am sure lots of people are having problems with this upstart issue and I feel this issue should be escillated to be fixed asap.

Issue also exists in kubuntu 9.10 as well.

Other than that everything else just works.

[sad1sm0]

This fixed my problems as well, but I have a question. When I do an apt upgrade it wants to upgrade upstart since it's been forced to a previous version. Is there a way to have aptitude ignore this package until the bug is fixed? I do regular system updates and I'd rather not have to force the version every single time.

[diddy1234]

you can hold upstart (once downgraded) by :-

To put a package to hold:
echo "upstart"|dpkg --set-selections

however, you will need to unhold the package at a later date.
This can be done by :-

to 'unhold' it:
echo "upstart"|dpkg --set-selecions

I have never done this myself but this should work.

[diddy1234]

Just performed a fresh install of ubuntu 9.10 then ran upgrades.
I then downgraded upstart to the previous version.
I then installed apache2 ssh and samba and configured them.

Now when i restart the server, the above installed and configured services do not start.

just wtf is going on?
surely there is a fundamental problem here.
I have never had collosal failiures like this before with ubuntu.

As a work around I can manually start the services each time I reboot the server and once up and running everything runs fine and is stable.
Oh well back to 9.04 I go !

[superyounan1]

I agree with Mr. Diddy, this is ridiculous. Karmic has been a big headache for me, and this is certainly one of the reasons.

[superyounan1]

I too downgraded upstart and things are working exactly as I would like again.

I see in Synpatic you can freeze a package version, and you can tell aptitude to do so too, but apt-get seems to ignore all that.

There are pages that explain 'pinning' packages in apt-get, but this seems a little screwy.

I found on this thread a mention of a utility called "wajig":
http://ubuntuforums.org/showthread.php?t=238706


Install wajig with:
apt-get install wajig

Then exclude the package with:
wajig hold <package>

Good Luck

I tried it out, and the results seem good, apt-get refrains from wanting to upgrade upstart:


The following packages have been kept back:
upstart
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

not sure how wajig did it, i'm assuming it pinned the package...

[ElHeineken]

A bug report has been created about this issue:
https://bugs.launchpad.net/ubuntu/+s...rt/+bug/500520 (https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/500520)

[superyounan1]

A bug report has been created about this issue:
https://bugs.launchpad.net/ubuntu/+s...rt/+bug/500520 (https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/500520)

fantastic, thanks

[jwvraets]

Thanks for the info. One puzzle is that on 2 virtually identical PCs sshd works (i.e.starts from boot) for one but not the other and I can see no differences. Most days I just love PCs - just not today.

[j23tom]

Check if you have loopback interface in:
/etc/network/interfaces

there should be (at file top):
auto lo
iface lo inet loopback

After I've added this, my services were back

[Eksilius]

Add me to the list of unfortunate ones. At least CUPS and vboxdrv fail to start at boot. The loopback thing wasn't it in my case. Considering to downgrade but I'm worried about possible dependency issues. Has anybody encountered such?

[ugutugut]

Solution: downgrade Upstart 0.6.3-11 to 0.6.3-10. Apache and Postgresql work normally now.

How? Using the Synaptic Package Manager search for upstart. Select it and then under Package click on Force Version. You get to choose between 0.6.3-11 and 0.6.3-10. Choose 10 and click on apply.

it's that easy. But it took me hours of googleing to find the solution. Very frustrating.

- Jon

Thanks Jon, your solution works for me. :D
It also fix my problem on Virtualbox, ie missing VBoxdrv and need to recompile every reboot.

[msabath]

Check if you have loopback interface in:
/etc/network/interfaces

there should be (at file top):
auto lo
iface lo inet loopback

After I've added this, my services were back

Hello to all.
I had the same problem with my services. After editing /etc/network/interfaces all the services work fine.

[jwvraets]

Check if you have loopback interface in:
/etc/network/interfaces

there should be (at file top):
auto lo
iface lo inet loopback

After I've added this, my services were back

j23tom, I did the above mentioned mod to /etc/network/interfaces and can confirm that on my system it does indeed do the trick (for ssh, cups and mysql that I have checked). Thanks ever so much. I owe you a beverage of choice if you ever get up this way.

JW

[Eksilius]

Tried downgrading upstart and not even that helped. I've tried everything suggested on the whole damn web but nothing works. ](*,) I'm getting close to wiping my disk and go for a clean install. That did it on my older computer, which had the same issue. I just don't like to think of the efforts.

[Eksilius]

At last, an update for Upstart showed up in the update manager today and I was of course extatic. Downloaded, installed, and - bummer again! I'm really going nuts here. Is a fresh install really the only way out of this? Or is even that going to help!? :x

[muckblit]

I installed lucid on five hard drives.

All installs came with /etc/network/interfaces auto lo:

auto lo
iface lo inet loopback

That is NOT a fix because upstart causes so many other problems. I think it's a fine concept but who released upstart 0.6.3-10 as the standard init method when it disables services like ssh and dbus, consequently no audio, maybe no X login? Would someone please make their bank accounts public ;-) ?

upstart 0.6.3.11 or newer does not run ifup -a. upstart never runs services such as dbus or gdm or kdm. No audio! This is killing ubuntu. Normally a bug of this strategic magnitude would be fixed in minutes, not hours, not days or weeks. upstart messes up boot logging so you can't see what's going on(research bootlogd and grub options quiet=1 and kernel opt quiet with NO splash). X is may not load or you may get a hung X load, frozen mouse, sluggish X. You may boot to a console login or just to hung X with frozen mouse.

I tried pinning upstart to what karmic is using:

#/etc/apt/preferences
Package: upstart
Pin: version 0.6.3-10
Pin-Priority: 900

Unfortunately a newer version of libc6 was already in place. apt-get or dpkg wanted to delete a hundred packages that depend on ANY libc6 in order to downgrade upstart and libc6. At least I was asked if I really wanted to do that, and of course I did not. If you are coming from karmic, you have upstart 0.6.3.10 so you have the opportunity to pin that before your upgrade. As long as you stay in karmic, you are frozen at upstart 0.6.3-10 until this too may pass.

It's possible that some install iso has a way. I did see dbus and audio working with upstart 0.6.3-11, but then ext4 got hosed some way before I could figure out the diff. I don't know if that lucid good upstart install was desktop or alt iso beta2 daily. I'd like to think somebody is making progress. It's only scripting, not hard if those familiar with upstart and udev and inittab/sysvinit would just try. I can't afford food, only coffee and gasoline, so don't lecture me about putting MORE time in on this.

[RikoW]

Hi all,

I stumbled across this thread when my cups service spontaneous didn't start anymore, but could easily started by hand. None of the tricks posted here help. Other problems appeared as well like cpu scaling not taking the default settings.

When I fiddled with the kernel and boot processes I started to use concurrency to speed up the boot process by running scripts in parallel on the 2 cores of my CPU. That seemed to have screwed it up. After I removed that setting, everything is working again as expected with the last available version on upstart.

Check the scripts


/etc/init.d/rc
/etc/default/rcS

for a line saying


CONCURRENCY=shell

If you find one, you might want to try and replace it with


CONCURRENCY=none

and reboot.

Cheers,

Riko

[Eksilius]

Yesss!

Finally, at last, something that works also on my computer! :D

But honestly, CONCURRENCY=shell is a long-running trick for speeding up boot on multi-core processors. I've even seen it in printed books in shops. And before applying it on my Karmic install (I've used it since Hardy), I did make sure the various web pages giving speed-tweak hints were still recommending it. And it did work fine until Upstart 0.6.3-11 came.

I definitely need to follow this up in bug #511387, which I issued. :confused:

[muckblit]

Upstart makes boot quicker than it was with sysv.

http://packages.ubuntu.com/karmic/upstart 0.6.3-10
http://packages.ubuntu.com/karmic-updates/upstart 0.6.3-11
http://packages.ubuntu.com/lucid/upstart 0.6.3-11build1

We can see that karmic is currently using upstart 0.6.3-10 which only has the worry of making sure /etc/network/interfaces has auto lo.

With karmic-updates in /etc/apt/sources.list, then the worry is that upgrading will move karmic users to upstart 0.6.3-11 and to a newer libc6 which(newer libc6) makes it very, very difficult to downgrade upstart because other newer packages depend then on newer lib6. Better karmic users pin their upstart to 0.6.3-10 for a while to keep from falling into the pit when /etc/apt/sources.list contains karmic-updates(synaptic shows checkmark for karmic-updates in repositories).

Sometimes upstart 0.6.3-11 works; sometimes not. I have a lucid 10.04 desktop live iso from January 12, 2010 (20100112.1) that has 0.6.3-11 and loads dbus, ssh, ifupdown eth0 OK, pulseaudio and gdm work because dbus is there. One could say that upstart 0.6.3-11 is not the problem! Somebody just has to do the scripting to come over to upstart from sysv. I may be able to tarball the /etc/init/ scripting that works there, and copy that over to my crippled no-service installations.

Some recent gdm scripting has gdm wait for dbus to load. That means our X will never start under gdm and the upstart no-services problem because dbus will never load.

http://www.mail-archive.com/lucid-changes@lists.ubuntu.com/msg04041.html
January 26, 2010 "convert openssh to upstart"...thus we don't have an upstart problem but only problems with services if they have not been revised to work with upstart!

[muckblit]

I used that Live iso from January 12 in which services start. It installs upstart 0.6.3-11. Services load, including gdm, cups, dbus, ssh. Then I pinned upstart version number before upgrading. Upgraded; services still load, but the upgrade installed upstart 0.6.3-11build1, in spite of a pin for 0.6.3-11. I had been assuming that -11build1 and -11ubuntu1 were "bad" and causing my no-service problem. Apparently it's the packages that depend on upstart that have to be fixed rather than 0.6.3-11 or -11build1 or previously assumed to be doomed -11ubuntu1.

gdm and openssh teams indicate on launchpad that they are working to adapt to upstart.

It's good to pin upstart_0.6.3-10 if you have just installed karmic but don't have karmic-updates(upstart_0.6.3-11) in your /etc/apt/sources.list or synaptic. 0.6.3-11 pin does not work, it upgrades to 0.6.3.11build1 anyway despite this pin:

#/etc/apt/preferences
Package: upstart
Pin: version 0.6.3-11
Priority: 900

...but 0.6.3-11build1 will work(services up) with the right install base even if that has 0.6.3-11.

About all I can do is poke around /etc/init to see if I can find diffs between 0.6.3.11* installs that do and don't work. Then maybe a tarball of a working /etc/init/ would fix broken situations! Other than that I seem to be just locating developer cliques that have released upstart-adapted deb packages for services. Outside looking in.

[dantonic]

Well downgrading upstart worked for me. thank you very much for the suggestion, I can finally use ssh the way I want to.

I am newer to ubuntu and linux in general.
I simply do not understand how or why a package like that could ever be put through. surely it must have been obvious that it broke so many features. What a waste of everyones time. Countless hours for me trying to fix these issues.

[citro_cell]

Check the scripts


/etc/init.d/rc
/etc/default/rcS

for a line saying


CONCURRENCY=shell

If you find one, you might want to try and replace it with


CONCURRENCY=none

and reboot.


Thank You!! Thank You!! After multiple reinstalls and searching through many different forums this is what fixed my problem!!

[citro_cell]

Thanks for all of the hints, I think I have combined them all for a solution. I had trouble holding the "Upstart" package with the following command
echo "upstart"|dpkg --set-selections
However after installing Wajig (tip from superyounan1) I was able to prevent "Upstart" from re-upgrading. Install Wajig with

sudo apt-get install wajig
Run Wajig

wajig
("JIG>" prompt appears) type:

hold upstart
and you are done. Hope this helps!! You can check to make sure that it worked by running update manager again and hopefully upstart will be grayed out.

[abumaia]

Hi all,

I stumbled across this thread when my cups service spontaneous didn't start anymore, but could easily started by hand. None of the tricks posted here help. Other problems appeared as well like cpu scaling not taking the default settings.

When I fiddled with the kernel and boot processes I started to use concurrency to speed up the boot process by running scripts in parallel on the 2 cores of my CPU. That seemed to have screwed it up. After I removed that setting, everything is working again as expected with the last available version on upstart.

Check the scripts


/etc/init.d/rc
/etc/default/rcSfor a line saying


CONCURRENCY=shellIf you find one, you might want to try and replace it with


CONCURRENCY=noneand reboot.

Cheers,

Riko

OMG YTMD! That finally fixed it for me, when every other tip and instruction had failed.

[benssig]

HOWTO DOWNGRADE UPSTART

(AND GET ALL YOUR STUFF WORKING AGAIN!!! - cron, samba, openvpn, etc...)


sudo aptitude install upstart=0.6.3-10

All my stuff was broke as a joke and now it's good to go.

:popcorn:

[hislitermos]

hi all,
i am newbie on ubuntu and using 9.10 version.
i am trying to use crontab function but failed.
when i type crontab -e i edit opened file but couldnt save
after editing, i typed CTRL+X button but ubuntu does not ask a question to save it.

how come can i save edited file ?
thanks in advanve



I think cron is working alright now. I had to re-install my local crontab file with:


crontab -eand then making a change and saving it. Still no luck on the other services. If I run:


sudo invoke-rc.d service restartthen the service will start correctly. I'd rather have this run automatically.

[BigEd55]

For what its worth... I have tried just about all the items listed above and some work at times, and at other times they don't. Nothing was reliable... Hit and miss until I came up with these steps. But the main thing I'm trying to figure... Since this has been around awhile and I've had this box up for awhile why did it wait to hit me until a week ago.

What I ended up doing was...
1- Reloaded the Upstart to the 0.6.3-11 version.
1a- Checked for any other upgrades (there was none)
1b- Reinstalled SSH and FreeNX with sudo apt-get install --reinstall {*********
** both 1a and 1b were done just to see if any entries were put into /etc/init (there were none)

2- Added these entries to rc.local (force them to start)
/bin/bash /etc/init.d/ssh start
/bin/bash /etc/init.d/freenx-server start

3- Un-Commented in /etc/ssh/sshd_config file the ListenAddress entries.
I added in this order ListenAddress 127.0.0.1 and followed on the next line with ListenAddress 0.0.0.0 in this order...

Noted that the interfaces still had the default entry the whole time
iface lo inet loopback

My opinion is that these services aren't up-to-date with the upstart, thus failing. And with the above changes I've reboot many times and still am able to get logged onto the box. Also noting that my server is thousands of miles away with no physical access unless I want to pay for someone to do it. I only have rescue mode access provided by my Provider.

[mohadib]

This bug stil exists today in server 10.04. Yet, according to this:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/216847

This bug is marked low priority. I can not understand this at all. How can something as CRITICAL as starting ssh not be a #1 pritority on a server distro. If you cant reach your server, then who cares what it can do. Its worthless.

Please make ssh start at boot. Having ssh listen on any ip is not a solution.

[renjithforever]

upstart is causing big troubles..cron fails..
i wanted to get cron running in the startup somehow..
tried rc.local ...didnt work..rc.local doesnt run in 10.10 ...no idea why..!

after googling... a lot... i found no solutions..finaly solved it myself.. :)

upstart does run some other jobs ..
find any such job..

initctl list
find any running job
i found that ssh was running..
so all u have to do is to add cron to ssh.conf

sudo vim /etc/init/ssh.conf
or

sudo gedit /etc/init/ssh.conf
add a line..towards the bottom

exec /usr/sbin/cron
restart... cron should have started:D

[urwen]

i had a problem trying to start mysql, and it simply solved doing a reinstall of upstart (package->mark for reinstall)