Plugin idea: Dynamic server catalog


Recommended Posts

What it'd do: maintain a catalog of all the content on the server - files, folder, shares etc. The catalog would be built on initial install. After that it would hook into the unRaid filesystem and get updated automatically on any file modification/access, without needing to run a separate script on schedule. Thus it'd be like updating parity in realtime vs doing it on schedule.

 

Features it'd provide to the user -

 

- instant search of any file/folder on the server: would list the locations (disk, share) where its located, without having to search

- if a disk goes offline, can generate a report of all missing content and all affected shares/folders

- various stats such as most used, age, size map, distribution

- possibly a realtime view of shares/disks with file access lighting up sections

 

The key to all this is automatic updates of the underlying catalog by hooking into the file system, which I believe is shfs. IMO this would be a very powerful addon. I don't know if APIs exist to enable this, I've never written a plugin, but am interested.

  • Upvote 1
Link to comment

Love the idea - since besides all the benefits you already mentioned, it could also replace the cache dir plugins, as all queries short of the actual files can be served from that catalog (like browsing the directory tree of shares without spinning up any array drives). The cache drive would be a candidate to store that catalog, and as that is typically a SSD drive this would be very fast as well.

Link to comment

Thought no one had seen this. Can I get some input from the regulars/unRaid authors if this is possible, and what it'd take. Yes it'd be more powerful than cache_dirs. Also this kind of info is very well suited to be kept in memory for instant access.

 

The more I think of this the more uses I can come up with - e.g. you keep a history of the catalog and you have an instant view of how your server usage has grown over time. 

Link to comment
On 7/19/2017 at 3:52 AM, Videodr0me said:

Love the idea - since besides all the benefits you already mentioned, it could also replace the cache dir plugins, as all queries short of the actual files can be served from that catalog (like browsing the directory tree of shares without spinning up any array drives). 

 

Thats not feasible when dealing with external clients. Only custom tailored queries could be directed to the catalog. Clients like windows systems wouldnt be able to be switched.

Link to comment
On 7/21/2017 at 3:50 AM, itimpi said:

I do not think this is that practical as it is likely to involve some fundamental changes within Linux core code to be that useful.   UnRAID tries to avoid that sort of change.

 

How is this a core change? In fact it'd be no change at all if the right api hooks exist. I looked into this a bit more and inotify seems to provide this info, and its part of slackware.

 

Are there any unRaid authors on this forum or is it just users? 

Link to comment
1 hour ago, MrCrispy said:

 

How is this a core change? In fact it'd be no change at all if the right api hooks exist. I looked into this a bit more and inotify seems to provide this info, and its part of slackware.

 

Are there any unRaid authors on this forum or is it just users? 

 

How is it not a core change? Kindly point out all the technical kernel hooks you feel are straight forward. Inotify does nothing for intercepting reads of files and replacing the contents with different data.That is what you're asking for, replacing core filesystem directory handling with something else.

 

The only possible place it might be put in place is the user filesystem "shfs". For all other accesses outside of /mnt/usr/, it would require making major fundamental changes to intercept and replace accesses for reads and shadowing writes. At this point you're replacing the existing user filesystem with your own.

 

I have worked in Linux and have done Kernel development way back in the day.

 

Yes, it is a core fundamental change.

Link to comment

Sorry I am not a Linux expert, let alone a kernel dev, and I read a bit more about inotify and take it back. It seemed at first glance to provide a callback for every file operation but its only for specific folders.

 

I don't know if Linux itself has this capability. e.g. NTFS has a journal - https://en.wikipedia.org/wiki/USN_Journal - which can be easily queried by apps and is used for this kind of functionality. Its used by many programs for things like versioning, file history etc. I believe OSX had similar features which were used by Time Machine, and the new APFS is probably much more sophisticated for this feature.

 

This seems like a pretty basic Linux feature - e.g. how do iotop/lsof etc operate?

 

I am not asking for something to intercept file read/write calls, examine the data etc. That would be a very low level filesystem driver. Just a way to get notified when a file is created/moved/deleted - basically whenever its metadata has changed. which is a LOT less frequent operation.

Link to comment

Actually Unraid already uses the FUSE filesystem as it's main core component. And one of the reasons to have a filesystem in userspace is to have flexibility in how to handle it's operations. @MrCrispy: You might look here https://github.com/libfuse/libfuse. Especially the passthrough example might give you a good starting point: https://github.com/libfuse/libfuse/blob/master/example/passthrough.c

But I guess you will need some support by the unraid dev team to make this work - it is far from a trivial undertaking.

Link to comment
  • 1 month later...
On 19.7.2017 at 9:52 AM, Videodr0me said:

Love the idea - since besides all the benefits you already mentioned, it could also replace the cache dir plugins, as all queries short of the actual files can be served from that catalog (like browsing the directory tree of shares without spinning up any array drives). The cache drive would be a candidate to store that catalog, and as that is typically a SSD drive this would be very fast as well.

 

Just curious... does SSD involve cloud computing?

Edited by carolyn6
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.