ntfs-3g allows illegal characters? You gotta be kidding...

[Eladon]

Since the beginning of computer time, people have known that the operating system needs to disallow certain characters in filenames when using Microsoft's file formats (such as ?, *, slashes, etc).

Well, I just wasted a ridiculous amount of time with a situation that was only getting worse each time I did something, only to realize that the original cause was that Ubuntu's implementation of ntfs-3g allows for characters that are traditionally considered illegal from Windows operating systems! #-o

A techno-geek type might be tempted at this point to start discussing about exactly what can and can't be done from Windows with filenames. And we could also chat about whether there's any point to GUI's because all of Linux can be controlled from the command prompt. But in the interest of considering Linux from the average guy's perspective, which is what I was hoping Ubuntu was all about, I personally just can't get over how unbelievably short-sighted this is. [-X

How did the illegal characters get there? Sometimes from copy/pasting files that originated on Linux partitions, sometimes from copy/pasting the filenames themselves, sometimes the names were from websites, and sometimes they were autogenerated from software (such as ripping music CD's). The point is, who cares where they came from, it shouldn't be MY problem to care, that's what I have an "Operating System" for! [/rant]

OK, I feel much better now (aside from the fact that I have hours of work ahead of me picking through my ntfs partition to try and find every single odd occasion of illegal characters in my files that Windows would have never allowed, but Ubuntu has).

All this being said, I can't believe that everyone actually tolerates this, so my question is this. Is there some kind of tweak or add-on that I can apply to make ntfs-3g actually work in a practical and safe way, you know, in the same way Microsoft uses ntfs, or for that matter how Ubuntu uses samba shares on networks?

In searching, I found the following links that appear to be in the correct context, but not practically useful:
https://bugs.launchpad.net/ubuntu/+bug/230906
http://ubuntuforums.org/showthread.php?t=622804
http://ubuntuforums.org/showthread.php?t=663807

[CatKiller]

From man ntfs-3g:


Windows Filename Compatibility
NTFS supports several filename namespaces: DOS, Win32 and POSIX. While
the ntfs-3g driver handles all of them, it always creates new files in
the POSIX namespace for maximum portability and interoperability rea‐
sons. This means that filenames are case sensitive and all characters
are allowed except ’/’ and ’\0’. This is perfectly legal on Windows,
though some application may get confused. If you find so then please
report it to the developer of the relevant Windows software.

[Eladon]

OK, fair enough, so since we're talking about applications and we're making quotes, let's talk about Microsoft's applications (http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx#basic_naming_conventions):

The following fundamental rules enable applications to create and process valid names for files and directories, regardless of the file system:
The following reserved characters are not allowed:
< > : " / \ | ? *
ROTFLMAO.

Dear Bill Gates:
According to the ntfs-3g developers, you didn't make Windows XP right. Yes, I realize that you are the creator and developer of both Windows and ntfs, and therefore it is you, and not others who decide what they do and don't do. And yes I realize that your documentation clearly explains that certain characters are illegal in Windows. And yes, I realize your developers represent over 90% of the market share, whereas the ntfs-3g developer's share of the pie is almost negligible and therefore their opinion of what should and shouldn't be standard practice is effectively meaningless in the real-world. But just the same, please fix Windows XP, as you clearly didn't write it correctly.

It's just like that time when Putin told an artist that he had to fix his painting (http://news.bbc.co.uk/2/hi/europe/8095960.stm). Except in this case it's more like some guy off the street who figures he's a good artist, telling some famous painter to fix his painting... I mean seriously?

It's strange, because one of the reasons I wanted to get away from Windows was because of the arrogancy of Microsoft. I was tired of this very thing, Microsoft telling me how I should and shouldn't use my computer. Now I have the ntfs-3g developers doing it instead. :confused:

On the other hand, if the claim is that ntfs and ntfs-3d are technically the same, than in actuality the real "software" problem is in Ubuntu's implementation of ntfs-3g, as they should be implementing ntfs-3g in the same way that windows implements ntfs, so that the two systems are effectively compatible. Doesn't anyone else think it's ironic that the entire point of ntfs-3g is compatibility, and yet it's not compatible?

Anyhoo, not sure what the quote proves, but since I have no sympathy for cats, you're cool in my books. 8)

[jocko]

Doesn't anyone else think it's ironic that the entire point of ntfs-3g is compatibility, and yet it's not compatible?
ntfs-3g is fully compatible with ntfs, but windows is not (that's ironic). The file system itself is case-sensive and allows some of the "illegal" characters (it accepts any unicode characters except /), it's just that microsoft haven't bothered to make their operating system support those characters, and even if the file system is case-sensitive, the operating system ignores the case completely except for the way it shows the filenames. Don't blame ntfs-3g. It just does what it is told to do, with the capabilities of the file system as limits.

But perhaps ntfs-3g should be configured for windows compatibility instead of full ntfs compatibility so it doesn't allow filenames that windows would not understand...

[Eladon]

Agreed.

And therefore:

1) Whether it's ntfs-3d's problem or not, regardless ntfs-3g is arrogant to state in their man page that it is the responsibility of Microsoft software to comply with their ideology. If you're going to make a driver who's whole point is compatibility with an industry standard, keep your personal non-industry-standard ideology out of your man page! Or if you insist on bringing up the subject, do the respectable thing and tell developers using your driver that it is their responsibility to implement their driver in a way which is compatible with Windows standards.
2) Ubuntu is responsible to implement ntfs-3d with restricted characters, just the same as Windows implements ntfs with restricted characters, and therefore this is effectively a bug in Ubuntu (since you can be sure if something were found like this in Windows, it would be considered a bug in Windows!!) If it is not possible for Ubuntu to do this reasonably, then the problem is still ntfs-3d, and therefore the truth is the ntfs-3d developers have made their so called compatibility software into a non-industry-standard extremist agenda, by forcing their ideology down our throats. If compatibility were their true objective, they would provide a way to use their driver in a way compatible with Windows.

Either way, it's just plain wrong.

[P4man]

The following fundamental rules enable applications to create and process valid names for files and directories, regardless of the file system:
The following reserved characters are not allowed:

Your quote says it all, doesn't it? Its an OS limitation, not a filesystem limitation. If windows one day can read/write HFS+ or even Ext3, should we ban those characters just so Windows doesn't get confused? Should we drop case sensitivity ? Drop all the security and permission features, just because windows can't use it?

Who ever made the filesystem or whatever their marketshare really seems irrelevant to me. MS made a decent filesystem, but their OS doesn't make full use of it.

I agree though with the previous poster that it might make sense to have a "windows compatibility mode" or something for NTFS. Cripple the filesystem to get along better with the crippled OS :)

[CatKiller]

The following fundamental rules enable applications to create and process valid names for files and directories, regardless of the file system:
The following reserved characters are not allowed:
< > : " / \ | ? *

So because some Windows applications might at some point be writing to a FAT drive, NTFS on Ubuntu should be crippled? That's insane. And asinine.

There is no problem using those characters on NTFS. No problem at all. But Windows applications can't assume without checking that the filesystem is NTFS, and most applications won't check, so the guideline for writers of simple applications, which is what you linked to, is to avoid those characters.

You say that you've known about Windows' limitations "since the beginning of computer time," so you should have noticed the filenames when you were copying-and-pasting files. You can't simultaneously be so arrogant about your knowledge of the subject and so ignorant of what you're doing and expect to be taken seriously. It's trivial to avoid those characters with a simple checkbox when ripping cd's (it may even be the default, I can't remember).

Anyway, there is a workaround described in the ntfs-3g documentation for those who want their NTFS partitions to be just as crippled as their FAT partitions.


Workaround: If case insensitivity handling and/or restriction of special character usage is desirable then you may export the NTFS volume via Samba (http://www.samba.org/) which supports this functionality the same way as it does for other POSIX file systems. An alternative solution for case insensitivity support is using ciopfs (http://www.brain-dump.org/projects/ciopfs/) which mounts a normal directory on a regular file system in case insensitive fashion.

[Eladon]

Catkiller, you'd make a good salesman for the "ntfs ultimate burger". You can feed it to your kids, you can feed it to your spouse, makes a great meal for the horse, the cow, the chickens, the dog or the cat, use it through your whole farm! Just think about how much more you can do with my new improved ntfs-ultimate-burger. Of course, it's not actually compatible with the human digestive system like the old burger was. But if you're reasonably intelligent, you'll pay very close attention and remember to eat every single bite with a spoonful of fiber, because otherwise you'll have to spend 3 hours on the toilet to get it out the other end. But just think of how much more you can do with it! Eating the regular old crippled burger that's actually compatible would simply be asinine! :D

Oh, and the developers are the "ntfs ultimate burger" scientists: Sorry folks, our burger is not compatible with the human digestive system. Please report to your doctors, because they should plan surgery to remove your intestines and replace them with our new super-pooper synthetic intestines, and you'll be good to go! Of course, you'll never be able to eat human food again, but isn't the ntfs-ultimate-burger great?

Nuff-said. ):P

[P4man]

You can make as many silly analogies as you want, it really doesn't help your argument. ntfs-3G implements the ntfs filesystem for linux, it doesn't implement all the (filesystem independent) windows OS limitations.

Are you arguing linux should not implement "? \>/... " characters or case sensitivity for fat32 as well?

[t0p]

As I'm basically a generous guy, I'm prepared to assume that the OP started this thread in good faith and actually believed that ntfs-3g doesn't follow the ntfs protocol.

But he has now been presented with evidence that ntfs-3g in fact does follow the ntfs protocol correctly, and it is XP's need to remain compatible with FAT that causes XP to treat allowable characters as illegal. His refusal to accept the facts and his puerile, trollish responses demonstrate that now his only wish is to denigrate ntfs-3g for an invented "incompatibility".

If I was a mod, I'd close this thread. Maybe one day I will be, and I will.

[3rdalbum]

I noticed this years ago, and I notified the developers.

Since then, I've come to see that there are points for and against it. The NTFS specification allows those characters. Linux allows those characters. If you have to borrow someone's NTFS-formatted flash drive to transfer files between two Linux machines, obviously you couldn't care less about the filename restriction.

It actually gets more complicated: You can theoretically use files with those "illegal characters" on Windows, but you can't create them, and many programs won't actually be able to work with them. But some can, depending on what part of the Windows API they use.

I still believe there should be a mount-time option for NTFS-3G that creates the restriction, but it's obviously something that the NTFS-3G developers have thought about and come to their own decision about. And you have to respect that.

[stefangr1]

I think the choice of the ntfs-3g developers to follow the ntfs protocol without paying attention to the limitations of operating systems isn't so strange. After all, their ntfs driver can have all sorts of applications, also on embedded devices.

However, with respect to the implementation of the ntfs-3g driver in Ubuntu, I must say I agree with the OP. I guess the use of ntfs-3g is to a large extent restricted to people with dual boot systems, external harddisks with ntfs that have to work with both linux and windows, etc. In that case, it's all about good compatibility with MS-windows...

Or am I wrong and are there really Ubuntu users that use ntfs-3g solely because they think ntfs is the best filesystem currently available for the linux platform?

[coffeecat]

Or am I wrong and are there really Ubuntu users that use ntfs-3g solely because they think ntfs is the best filesystem currently available for the linux platform?

Not quite, but I do use NTFS-formatted drives and the ntfs-3g driver in Ubuntu for one slightly ironic reason: the lack of Unix permissions support in NTFS. As a single user of my systems, where I like to try out other distros as well, I find that Linux/Unix file ownership can get in the way. Try plugging an ext2/3 formatted drive into your system. It gets mounted root-owned and you have to go through hoops reading/writing your own files. Try swapping files between a Mac and an Ubuntu system using the hfs+ filesystem (Ubuntu/Linux can only write to the unjournalled version anyway) and you get ownership problems because of a differing UID. Try swapping files between your Ubuntu partition and your Fedora (for example) partition and you get ownership problems - also because of a different UID.

But use NTFS and these problems melt away. You can even install the ntfs-3g driver on your Mac to obviate the 4GB file limit with FAT32 when swapping files between Linux and MacOS. The one serious issue is that you really need Windows around to repair an NTFS filesystem with chkdsk if it ever gets corrupted.

[asmoore82]

ROTFLMAO.

Bill Gates is the creator of NEITHER, NOR.
Neither Windows,
Nor WinNT,
Nor DOS,
Nor NTFS.

WinNT and NTFS were and always will be just a poor attempt to emulate UNIX.

[t0p]

However, with respect to the implementation of the ntfs-3g driver in Ubuntu, I must say I agree with the OP. I guess the use of ntfs-3g is to a large extent restricted to people with dual boot systems, external harddisks with ntfs that have to work with both linux and windows, etc. In that case, it's all about good compatibility with MS-windows...


CatKiller already described above how you can ensure compatibility with Windows OS by exporting the ntfs volume via Samba or by using ciopfs. So we can have a proper implementation of ntfs according to its specification and we can be sure that file names etc will obey the limitations required by XP if necessary. So everybody's happy. Or nearly everybody anyway. Where's the problem with that?

[asmoore82]

Agreed.

And therefore:

1) Whether it's ntfs-3d's problem or not, regardless ntfs-3g is arrogant to state in their man page that it is the responsibility of Microsoft software to comply with their ideology. If you're going to make a driver who's whole point is compatibility with an industry standard, keep your personal non-industry-standard ideology out of your man page! Or if you insist on bringing up the subject, do the respectable thing and tell developers using your driver that it is their responsibility to implement their driver in a way which is compatible with Windows standards.
2) Ubuntu is responsible to implement ntfs-3d with restricted characters, just the same as Windows implements ntfs with restricted characters, and therefore this is effectively a bug in Ubuntu (since you can be sure if something were found like this in Windows, it would be considered a bug in Windows!!) If it is not possible for Ubuntu to do this reasonably, then the problem is still ntfs-3d, and therefore the truth is the ntfs-3d developers have made their so called compatibility software into a non-industry-standard extremist agenda, by forcing their ideology down our throats. If compatibility were their true objective, they would provide a way to use their driver in a way compatible with Windows.

Either way, it's just plain wrong.

How much have you paid to license Micro$oft's defective OS,
that's self-admittedly incompatible with its own default filesystem??

How much have you paid to the ntfs-3g developers for their 100% correct and working software?

No one here is forcing anything on you - you won't be forced to use Ubuntu or ntfs-3g.
If it's not something you find useful, then by all means don't use it.
Complaining endlessly about how the developers are all wrong but you are right "just because" and
praising the likes of Micro$oft, Windows, and Bill Gates will get you absolutely nowhere, and fast too.

You are just plain wrong. Have a nice day.

P.S. "Windows standards" is a contradiction in terms and shows massive ignorance on the part of anyone who says it.

[asmoore82]

Not quite, but I do use NTFS-formatted drives and the ntfs-3g driver in Ubuntu for one slightly ironic reason: the lack of Unix permissions support in NTFS. As a single user of my systems, where I like to try out other distros as well, I find that Linux/Unix file ownership can get in the way. Try plugging an ext2/3 formatted drive into your system. It gets mounted root-owned and you have to go through hoops reading/writing your own files.
:shock: #-o :confused: ](*,)
That is truly awe-full.

ext3 most certainly does not get mounted "root-owned" - it stores UNIX ownership/permissions properly.


$ ls -ld /mnt/sda3
drwxr-xr-x 2 root root 4096 2009-05-11 00:32 /mnt/sda3
$ sudo mount /dev/sda3 /mnt/sda3
$ ls -ld /mnt/sda3
drwxr-xr-x 4 root root 4096 2009-08-31 15:55 /mnt/sda3
$ sudo chown asmoore /mnt/sda3
$ ls -ld /mnt/sda3
drwxr-xr-x 4 asmoore root 4096 2009-08-31 15:55 /mnt/sda3
$ sudo umount /mnt/sda3
$ ls -ld /mnt/sda3
drwxr-xr-x 2 root root 4096 2009-05-11 00:32 /mnt/sda3
$ sudo mount /dev/sda3 /mnt/sda3
$ ls -ld /mnt/sda3
drwxr-xr-x 4 asmoore root 4096 2009-08-31 15:55 /mnt/sda3

[CatKiller]

WinNT and NTFS were and always will be just a poor attempt to emulate UNIX.

Actually, NT was quite a deliberate (though still arguably poorly implemented) attempt to re-create UNIX. It was written by developers from DEC largely to compete with Novell, and to get application developers away from UNIX. It still has a POSIX compatibility layer (Services for Unix, IIRC).


ext3 most certainly does not get mounted "root-owned" - it stores UNIX ownership/permissions properly.

I think coffeecat's point was that permissions are controlled by UID number. So if you have a bunch of files owned by user 1001 on one system, and then put those files in a different system where user 1001 is a different user, all those files will be owned by that second user instead. Which is a valid scenario in an ad-hoc arrangement where you haven't paid particular attention to which user gets which UID. Using NTFS to nullify all permissions, though, is... novel... as an approach. :)

[coffeecat]

ext3 most certainly does not get mounted "root-owned" - it stores UNIX ownership/permissions properly.

Yes, of course it stores Unix ownership/permissions properly. That's the point. Format a USB drive ext2/3, and then plug it in to your Ubuntu machine. It will be automounted, an icon will appear on your desktop and a Nautilus window will open. Now try dragging and dropping one of your files across to the Nautilus window. That's what I mean. It's not particularly difficult to get round, but it's irritating, and certainly not newbie-friendly, and you don't get that if you plug a USB ntfs drive in.

Compare also with what happens when you plug a USB drive formatted HFS+ into a Mac and you drag and drop files into it. That's the level of GUI user-friendliness that we are not getting in Ubuntu with ext2/3 formatted USB drives.

Or at least we weren't last time I looked. :wink:

[Slim Odds]

But he has now been presented with evidence that ntfs-3g in fact does follow the ntfs protocol correctly, and it is XP's need to remain compatible with FAT that causes XP to treat allowable characters as illegal. His refusal to accept the facts and his puerile, trollish responses demonstrate that now his only wish is to denigrate ntfs-3g for an invented "incompatibility".

I've really grown to enjoy unrealistic expectations... but, then again, I've been married a long time......

[rygle]

Some good news on this bug. I put up a comment on the NTFS-3G forums and got a very positive reply from Jean-Pierre, one of the lead programmers for NTFS-3G;

I asked...


Could you please include a switch in NTFS-3G to allow a workaround for this Windows limitation? (i.e. to turn off use of special characters in order to allow compatibility with Windows for the many users who wouldn't have a clue how to re-mount/export their NTFS filesystems using Samba)


And Jean-Pierre replied...


This is available in the release candidate advanced ntfs-3g-2010.5.16AR.1, see http://pagesperso-orange.fr/b.andre/advanced-ntfs-3g.html
The option to use is "windows_names", documented in the manual of the said version.


Hopefully this will help a lot in this area, but it would need to be integrated into Ubuntu by someone cleverer than me...

This discussion can be found on the Tuxera site - http://tuxera.com/forum/viewtopic.php?f=2&t=763&p=14549&sid=053f5d02fb19e22aeda1019f6225c76d#p14549

[rygle]

BTW, this issue is mentioned under at least two Ubuntu bug reports. I have also commented about this development there.

https://bugs.launchpad.net/ubuntu/+source/rhythmbox/+bug/318625
and https://bugs.launchpad.net/ubuntu/+bug/230906