fredsherbet

Even uncached, I don't think it should take that long. Music contains an iTunes library (including movies and tv shows), so the majority of files are 4 or 5 directory levels deep. I don't have the cache_dirs scripts. I'll look in to it.

Thanks for taking the time to help me Repeating the command is quick after the initial slow one. All disks are spun up before I run the command. System has 1GB memory. root@centax:~# free -m total used free shared buffers cached Mem: 978 925 52 0 179 455 -/+ buffers/cache: 291 686 Swap: 0 0 0 I have 1 parity (2TB) drive, and two data drives (1TB, 1.5TB) I have two user shares, but both are 'basic' - no includes/excludes. High water. As far as I can tell, my disks are in good condition and happy. However, both data drives are about 90% full. Here's a couple more tests, which I'd expect to execute in a few seconds. root@centax:/mnt/user/matthew# time du -sm Music 710521 Music real 7m44.589s user 0m0.250s sys 0m1.560s root@centax:/mnt/user/matthew# find Music/ | wc -l 47311 root@centax:/mnt/user/matthew# time du -sm LivingRoom/ 77825 LivingRoom/ real 1m48.380s user 0m0.060s sys 0m0.470s root@centax:/mnt/user/matthew# find LivingRoom/ | wc -l 11930

The test isn't good evidence, but I've found du -sm to be consistently slow on my unraid for the last month or so. I'm not sure about before then. It can take several hours on a large folder (e.g. ~300GB of varying size files) but I've never found it to take that long on similar size folders on my main computer. My unraid is running on Atom, but it's hardly hitting the CPU/memory at all when doing these operations.

Hi I think du is taking a long time on my unraid (I also think that solving this will make my backup much faster using rsync) Here's the results of a test, on a folder that contains 6000 files, totaling 1735MB On my main computer (not unraid) it takes 0.3s. On unraid it takes 31 seconds Main computer (not unraid) Coruscant:~ matthew$ time du -sm Backgrounds.aplibrary/ 1735 ./Backgrounds.aplibrary/ real 0m0.284s user 0m0.013s sys 0m0.183s unraid root@centax:/mnt/disk2/matthew/Pictures/Libraries# time du -sm Backgrounds.aplibrary/ 1737 Backgrounds.aplibrary/ real 0m31.785s user 0m0.070s sys 0m0.160s Is that expected? Or is there something wrong with my box? Cheers!

Thanks for the extra infos

Hi, I thought I'd share my success getting rsync faster. I almost doubled the transfer speed. I found that the bottleneck was SSH's CPU usage on the target box, due to decrypting being expensive. You can improve performance easily by using blowfish encryption, which is less secure than 3DES, but significantly less computationally expensive. Put this in your rsync to make that happen: rsync --rsh="ssh -c blowfish" .... This got me from 15MB/s to about 20-25MB/s. I got more gain by ditching ssh for rsh. RSH is completely insecure. Do not use it over the open Internet. If you're on a private home network, then it completely removes the SSH overhead. Getting it to work on unraid took a little fiddling. You can get an rshd package from http://pkgs.org/slackware-13.1/slackware-i486/netkit-rsh-0.17-i486-1.txz.html Edit /etc/inetd.conf to uncomment the rsh.d (starts shell) line, and run kill -HUP `pgrep inetd` so inetd reloads its config, and knows to listen for rsh connections, and call rshd to handle them. Edit ~/.rhosts (create it, if it doesn't exist) to list the IP addresses that are allowed to connect to the system. When debugging, tail /var/log/syslog to see what's not working. Using RSH got me from 15MB/s to 25-30MB/s.

4.6

Okay, so stepping back and looking at the actual issues I was hoping to resolve with iSCSI: iTunes can't complete a 'consolidate' library operation - it gets going, and then reports a required file cannot be found. I'm running it now, consolidating from the unRAID to a local disk, and it's going fine. The trouble with Aperture is that it doesn't recognise the unRAID volume is the same volume every time. I have the library on a local disk, with referenced masters on unRAID. Most of the time this is fine, but if I start Aperture with the unRAID volume connected, it declares the masters offline (fine) but then won't recconnect the masters when the volume is online again (with or without a restart of Aperture) I have to manually point out where they've 'moved' to. It takes it quite a long time to process all the photo locations again. I think the problem is related to some volume ID. Interestingly, I recently noticed that when I run df on the mac, the AFP filesystem has a changing value: afp_00yVhe00yVhe00yVhe00yVhe-1.2e000007 4883653136 3762084144 1121568992 78% /Volumes/matthew The 000007 number increments after each connection.

I've got AFP on my unRAID (with netatalk, so it appears like magic in Finder). I don't think it improved things compared to samba. I think AFP does a better job with permissions and properties, but that wasn't having an impact for me.

What I was hoping to solve with iSCSI was instability using Apple apps with AFP or Samba. The connections seem stable - they don't drop or disappear at all, but some Apple software that heavily uses the link (iTunes and Aperture) doesn't always work the way I'd like. Aperture forgets that the unRAID volume is where the master pictures are. iTunes seems to lose connection with its files (on unRAID), and has in the past deleted apps/music from my iPhone, and can't complete a 'consolidate media files' operation. I'm going to try using sshfs, which I had forgotten I have installed and working. The final solution will be to stop using unRAID as a primary media server, and just for backup and secondary media serving.

I did some digging and it would appear that is not correct. See http://lime-technology.com/forum/index.php?topic=1442.msg9894;topicseen#msg9894 shfs is some propriety lime-tech magic 'shared file system'. Is it not a block device? Is there some similar magic I can do, to create a virtual disk that my mac can read? (I admit, this is both clutching at straws, and probably not even a good solution)

Thanks, Joe. I've updated the original post with your clarification.

I think this is probably true It's been interesting to try, though Thanks for the info Aaron, I'll fix up those LUNs. I'm interested in what the difference is between the /dev/sdxx /dev/mdx and shfs 'devices' are. I know that the sdxx are the actual disks. The mdx are something to do with unRAID's parity array system, and shfs is something to do with SSHFS (just unsecured?) How come iSCSI can connect to the mdxs, but not shfs? Can I create some sort of virtual disk/file system that would work with OSX? Maybe a HFS+ partition inside the unRAID array?

I've posted a guide at http://lime-technology.com/forum/index.php?topic=10089.0 Note that it's not turning out particularly useful to me, yet! So check your requirements against what I think is possible (as described in that thread)

I've gotten iSCSI working well enough for unRAID to be an iSCSI target. However, the machine you connect from needs to be able to read/write reiserFS volumes, but you can test using another HDD in the unRAID, or using the flash drive. iSCSI is quite different from AFP and Samba - it isn't a network file sharing protocol; it lets a remote device connect to a disk (not a filesystem) on the target device (maybe?) as though the disk was connected locally. You can partition and format drives over iSCSI, but that also means the device you connect from has to recognise the file system on the drive. (EDIT:) Note - writing to the /dev/mdx devices won't break unRAID's parity, where as creating targets to the sdxx devices (e.g. /dev/sda or /dev/sda1) bypasses the unRAID array, and so will break parity. DO NOT CONNECT iSCSI TO /dev/sdX DEVICES. DISCLAIMER - I only understood about 75% of what I did. Don't blame me if anything goes wrong! The installation wasn't easy, but the most difficult bit was building the iSCSI install package. Here's the one I built. It's built for unRAID 4.6 on i686. unRAID 4.6's kernel is 2.6.32.9. http://fredsherbet.com/unraid/iscsitarget-1.4.20.2-i686-1.tgz References These are the references I used to set up a slackware dev box, build iSCSI target, install on unRAID, and configure it. * Target - http://iscsitarget.sourceforge.net/'>http://iscsitarget.sourceforge.net/ o Config http://manpages.ubuntu.com/manpages/hardy/man5/ietd.conf.5.html Building them requires kernel source and complicated stuff. Have to build on a full slackware install, then package up to install on unRAID. * http://lime-technology.com/forum/index.php?topic=2937.0 * http://www.linuxpackages.net/howto.php?page=package&title=Package+Howto Process I went through For reference, here's my rough notes for the whole process I went through. I had unRAID 4.6, which uses kernel 2.6.32.9. 1. Install Slackware in Virtual Box. 2. Get kernel source that matches the kernel on the unRAID box. 1. Download from ftp://ftp.kernel.org/pub/linux/kernel/v2.6/ ( wget ) 2. unpack (tar xvf) 3. copy directory to /usr/src 3. Set up the kernel source's config 1. Copy .config from /usr/src/linux on the unraid box to the kernel source directory on the slackware dev box 2. From the kernel source dir, run make oldconfig and answer the questions. 4. Build the kernel and modules (run make ) 5. Switch to a safe location (e.g. /root) and get iscsitarget ( wget the latest tar.gz from http://iscsitarget.sourceforge.net) 6. Untar it ( tar xvf ) 7. CD into the dir it creates and run make KSRC=/usr/src/linux-2.6.32.9 to build it. 8. Run make KSRC=/usr/src/linux-2.6.32.9 ARCH=i686 install DESTDIR=/work/install to install pretending destdir is the root. 9. CD to the /work/install dir and run makepkg to build the slackware package for unraid. 10. Copy to unraid, and run installpkg to install iscsi on the unraid box. 11. Edit the config file at /etc/iet/ietd.conf 12. Start the iscsi target by running /etc/rc.d/iscsi-target start (It's stopped again by running /etc/rc.d/iscsi-target stop) 14. Use globalSAN iSCSI to connect from a Mac. 1. Add a target. The target name is the Target option in the iscsi target config file. 2. Add a connection (just enter the unraid's ip address) 3. Click connect. 15. I could only connect to unraid's flash drive, since that's the only file system with a format the mac could recognise. See http://lime-technology.com/forum/index.php?topic=10082.0