fuse-music.git
6 years agoExpand the directory-tracking pre-coding logic master
Steve McIntyre [Sat, 1 Jun 2013 18:55:22 +0000 (19:55 +0100)]
Expand the directory-tracking pre-coding logic

The old code used to just track the single previous directory. This
would be thrown by multiple users reading files from different
directories, stopping parallelism from happening. Now, track the last
10 directories accessed all the time, and trigger if there are
sufficient matches in any of those positions. Much faster.

6 years agoHandle "quality" setting in the config file
Steve McIntyre [Sun, 26 May 2013 22:55:01 +0000 (23:55 +0100)]
Handle "quality" setting in the config file

Thinko previously ignored whatever was set in the config file and
always used the default value. Now fixed.

7 years agoAdd encode_ogg and encode_ogg_client scripts
Steve McIntyre [Sun, 19 Feb 2012 19:22:15 +0000 (19:22 +0000)]
Add encode_ogg and encode_ogg_client scripts

Simple but functional shell wrappers for encoding, ready to use with
fuse-music.

 * encode_ogg runs on the machine where fuse-music is installed

 * encode_ogg_client (optionally) runs on other machines on the
   network to spread the encoding load for faster performance.

7 years agoAdd trivial DB conversion tool
Steve McIntyre [Tue, 22 Nov 2011 18:56:19 +0000 (18:56 +0000)]
Add trivial DB conversion tool

Moved the location of all the flacs on my server to add an extra level
of directories. Rather than a complete re-encode, simply open the
database and modify all the flac paths to match.

7 years agofactor out $(CC)
Steve McIntyre [Tue, 22 Nov 2011 18:55:53 +0000 (18:55 +0000)]
factor out $(CC)

7 years agoPeriodically check the sizes db and remove obsolete entries
Steve McIntyre [Tue, 22 Nov 2011 18:51:18 +0000 (18:51 +0000)]
Periodically check the sizes db and remove obsolete entries

 * Spawn another thread at system startup.
 * Once per hour, that thread will crawl the DB and check for dead
   entries

7 years agoRe-factor and cleanup:
Steve McIntyre [Tue, 22 Nov 2011 18:41:45 +0000 (18:41 +0000)]
Re-factor and cleanup:

 * never use free_results() inside an _unlocked function, make the
   lock and data safety tracking easier
 * factor out db_remove_size_entry_unlocked() and
   db_remove_cache_entry_unlocked() for easier use inside other
   database functions

7 years agoFix silly thinko in db_dump_size_unlocked
Steve McIntyre [Tue, 22 Nov 2011 00:08:58 +0000 (00:08 +0000)]
Fix silly thinko in db_dump_size_unlocked

7 years agoFix for cache startup crash.
Steve McIntyre [Sun, 7 Aug 2011 21:15:14 +0000 (22:15 +0100)]
Fix for cache startup crash.

When syncing up the cache and DB at startup, be extra careful about
data safety. We haven't gone into normal code yet and the database
state might be interesting!

If we delete/modify an entry from the cache db, restart the entire
loop.

7 years agoFill in st_blocks so that du works
Steve McIntyre [Wed, 29 Jun 2011 23:59:54 +0000 (00:59 +0100)]
Fill in st_blocks so that du works

7 years agoAdd traces to the FUSE entry points
Steve McIntyre [Wed, 29 Jun 2011 23:59:17 +0000 (00:59 +0100)]
Add traces to the FUSE entry points

7 years agoAdd config file support
Steve McIntyre [Wed, 29 Jun 2011 23:25:05 +0000 (00:25 +0100)]
Add config file support

Supports simple text file config, using lines of the form

VAR = value

The code will deal with extraneous whitespace and commented lines
using "#"

All the normal fuse-music command line options may be added
here. FUSE-specific options (e.g. -d or the mountpoint) must still be
provided on the command line as normal.

7 years agoFix cut and paste error
Steve McIntyre [Wed, 29 Jun 2011 23:23:38 +0000 (00:23 +0100)]
Fix cut and paste error

7 years agoSwitch to strings for all the core option parsing
Steve McIntyre [Wed, 29 Jun 2011 17:20:02 +0000 (18:20 +0100)]
Switch to strings for all the core option parsing

Cleans up the macro handling in mount_opts and fm_mount_opts, ready
for adding config file support. The values of the two numberic config
options (cachesize and num_threads) are now split out into separate
file-global variables. We parse the strings once the fuse lib has
given them to us, then use the numeric variables from then on.

Allows us to use more user-friendly inputs like cachesize=10G (etc.)
easily in the program.

7 years agoUse xstrtoumax() for parsing strings into numbers
Steve McIntyre [Wed, 29 Jun 2011 17:17:21 +0000 (18:17 +0100)]
Use xstrtoumax() for parsing strings into numbers

Add wrapper functions in misc.c for parsing strings. Makes it possible
to parse cache sizes as "cachesize=10G", a lot more user-friendly than
"cachesize=10485760000".

7 years agoAdd xstrtol and friends from gnulib.
Steve McIntyre [Wed, 29 Jun 2011 17:15:55 +0000 (18:15 +0100)]
Add xstrtol and friends from gnulib.

7 years agoMore preprocessor magic to reduce repeated code
Steve McIntyre [Wed, 29 Jun 2011 14:33:19 +0000 (15:33 +0100)]
More preprocessor magic to reduce repeated code

7 years agoImproved logic for matching:
Steve McIntyre [Mon, 20 Jun 2011 01:49:30 +0000 (02:49 +0100)]
Improved logic for matching:

Used to simply look for any regexp (substring) match, causing issues
with common words used as track names.

Now:

  1. See if we have a complete and exact match.
  2. If we don't then fall back to the fuzzy substring match

7 years agoAdd extra debug
Steve McIntyre [Mon, 20 Jun 2011 01:48:47 +0000 (02:48 +0100)]
Add extra debug

7 years agoAdd extra debug
Steve McIntyre [Mon, 20 Jun 2011 01:47:44 +0000 (02:47 +0100)]
Add extra debug

7 years agoAdd missing semicolons
Steve McIntyre [Mon, 20 Jun 2011 01:45:39 +0000 (02:45 +0100)]
Add missing semicolons

7 years agowhitespace fixes
Steve McIntyre [Mon, 20 Jun 2011 01:44:51 +0000 (02:44 +0100)]
whitespace fixes

7 years agoSilly bugfix for exec() call
Steve McIntyre [Mon, 20 Jun 2011 01:42:06 +0000 (02:42 +0100)]
Silly bugfix for exec() call

The first entry in the list is used as argv[0] for the spawned
command, so need to add something there. May as well add
"sox". Previously, was losing the first flac file on the list here.

8 years agoadd wodim usage help
Steve McIntyre [Fri, 17 Jun 2011 16:26:26 +0000 (17:26 +0100)]
add wodim usage help

8 years agoSimple script to generate wav/cue files
Steve McIntyre [Fri, 17 Jun 2011 16:23:34 +0000 (17:23 +0100)]
Simple script to generate wav/cue files

Assuming a set of flac files and a matching cue file, this will use
the flac files and the cue file to make a new cue file and a wav file
of the music data ready for writing to CD.

Depends on use Audio::FLAC::Header (aka libaudio-flac-header-perl).

8 years agoDon't need to call free_results() in db_remove_cache_entry()
Steve McIntyre [Sun, 15 May 2011 22:12:43 +0000 (23:12 +0100)]
Don't need to call free_results() in db_remove_cache_entry()

8 years agoDon't need to call free_results() in db_store_cache_entry()
Steve McIntyre [Sun, 15 May 2011 22:08:01 +0000 (23:08 +0100)]
Don't need to call free_results() in db_store_cache_entry()

8 years agoChange how the worker threads are started so daemon() works
Steve McIntyre [Sun, 15 May 2011 21:31:05 +0000 (22:31 +0100)]
Change how the worker threads are started so daemon() works

Previously, things would only work properly when run in the
foreground. The reason for that was that the daemon() call broke links
between the main thread and threads already started. Move the thread
startup code into a helper function init_threads(), and use
pthread_once to make sure it's called once by whichever of the main
fuse interface functions gets called first.

8 years agoMore debug cleanup
Steve McIntyre [Sun, 15 May 2011 03:24:30 +0000 (04:24 +0100)]
More debug cleanup

8 years agoClean up default log levels
Steve McIntyre [Sun, 15 May 2011 03:13:45 +0000 (04:13 +0100)]
Clean up default log levels

8 years agoFix silly logic inversion in add_todo_entry_dir()
Steve McIntyre [Sun, 15 May 2011 02:58:43 +0000 (03:58 +0100)]
Fix silly logic inversion in add_todo_entry_dir()

When checking if a file is a flac or not (for ignoring purposes),
forgot to invert the test.

Added extra debug for the selection logic here.

8 years agoRe-instating mistakenly deleted line
Steve McIntyre [Sun, 15 May 2011 02:45:22 +0000 (03:45 +0100)]
Re-instating mistakenly deleted line

8 years agoFix potential race in entry_file_thread()
Steve McIntyre [Sun, 15 May 2011 02:28:54 +0000 (03:28 +0100)]
Fix potential race in entry_file_thread()

There's a potential for readahead to do more work than necessary - if
we're scanning a directory then we can end up issuing re-encodes for
files that have *just* been finished. Not a major problem, but
wasteful of CPU. At the very last point before enqueuing an encode for
a given file, check to see if:

 a) we're doing readdir (i.e. marked with ENCODE_DONT_KEEP)
 b) the file already has a valid record in the size DB

if these are both true, just don't re-encode. To allow this, we have
to pass the "keep" parameter around more.

8 years agoMajor updates for cache handling
Steve McIntyre [Sat, 14 May 2011 12:42:53 +0000 (13:42 +0100)]
Major updates for cache handling

Don't cache the cache size in fmcache.c any more - moved to the
database code, accessed via cache_current_usage() instead now. Works
much better this way, and we don't get out of step with the database.

Rename the locks used in the cache code, and wrap the locking calls
via LOCK() and UNLOCK() which can be compiled to log locks when
debugging.

When busy in the cache, there was a possibility that a read() call
could cause a file to be encoded and then the file could be dropped
from the cache again before we actually got a chance to read the
file. Fix that by adding extra state - if we're encoding a file to
service a read request, then pass a flag ENCODE_KEEP down the
stack. When we're done encoding that file, we will mark it in the
cache DB directly as FMDB_CACHE_READING instead of
FMDB_CACHE_FREE. That will cause the cache code to leave it
unmolested.

In fuse-music.c, make the debug output of dump_global_queue() much
easier to follow. Also tweak the logic when allocating cache space for
an encode operation. Iff we already have a size entry for a given
track, then we ask for that size instead of the flac size.

Fix thread startup:
 * use pthread_attr_init() before trying to set thread attributes to
   make sure we have a clean initial state.
 * Increase the thread size to 4MB

8 years agoMajor updates and cleanups to the cache DB code
Steve McIntyre [Sat, 14 May 2011 12:33:54 +0000 (13:33 +0100)]
Major updates and cleanups to the cache DB code

Keep a local cache of the amount of cache space used. Update this
after every write to the cache DB that may modify this, and expose it
via the new interface function db_cache_space_used()

db_count_cache() now returns the number of cache entries, useful
later.

Also moved dump_size_entry() to db_dump_size_entry(), allowing for
external use.

8 years agoSwitch the XXXLOGx commands to using the LOGPRINT macro
Steve McIntyre [Sat, 14 May 2011 12:22:44 +0000 (13:22 +0100)]
Switch the XXXLOGx commands to using the LOGPRINT macro

Used to use fprintf directly; now LOGPRINT locks and unlocks the
logfile for each sequence so we don't get interleaved writes.

8 years agoMove from using fprintf to using MAINLOGx
Steve McIntyre [Sat, 14 May 2011 12:21:16 +0000 (13:21 +0100)]
Move from using fprintf to using MAINLOGx

More configurable debug

8 years agoAdd generic LOGPRINT macro helper which locks
Steve McIntyre [Wed, 11 May 2011 20:25:57 +0000 (21:25 +0100)]
Add generic LOGPRINT macro helper which locks

8 years agoIf the DB fails unexpectedly, dump its contents
Steve McIntyre [Thu, 27 Jan 2011 00:39:09 +0000 (00:39 +0000)]
If the DB fails unexpectedly, dump its contents

8 years agoExpose the db_dump_* functions externally via wrappers
Steve McIntyre [Thu, 27 Jan 2011 00:36:39 +0000 (00:36 +0000)]
Expose the db_dump_* functions externally via wrappers

8 years agorearrange locks in the cache code
Steve McIntyre [Thu, 27 Jan 2011 00:26:00 +0000 (00:26 +0000)]
rearrange locks in the cache code

8 years agoAdd finer-grained control of debug
Steve McIntyre [Thu, 27 Jan 2011 00:07:48 +0000 (00:07 +0000)]
Add finer-grained control of debug

8 years agoremove unused variable
Steve McIntyre [Wed, 26 Jan 2011 23:59:08 +0000 (23:59 +0000)]
remove unused variable

8 years agoUse mtime properly in the cache DB
Steve McIntyre [Wed, 26 Jan 2011 23:37:40 +0000 (23:37 +0000)]
Use mtime properly in the cache DB

Rather than just storing the mtime of the underlying flac, store the
time we last used the cache file itself.

8 years agoCall free_results() more consistently
Steve McIntyre [Wed, 26 Jan 2011 23:37:06 +0000 (23:37 +0000)]
Call free_results() more consistently

8 years agoExtra DB debug functions and cleanup
Steve McIntyre [Mon, 24 Jan 2011 01:47:46 +0000 (01:47 +0000)]
Extra DB debug functions and cleanup

Rearrange order of functions: keep cache and size functions together.

Add dump_size_entry(), db_dump_size() and db_count_size() for future
debug purposes.

8 years agofix up size handling, go unsigned to signed
Steve McIntyre [Mon, 24 Jan 2011 01:23:37 +0000 (01:23 +0000)]
fix up size handling, go unsigned to signed

8 years agoAdd db_count_cache function to help in debugging
Steve McIntyre [Mon, 24 Jan 2011 01:21:16 +0000 (01:21 +0000)]
Add db_count_cache function to help in debugging

8 years agoAdd more debug in the LRU path
Steve McIntyre [Sun, 23 Jan 2011 21:56:57 +0000 (21:56 +0000)]
Add more debug in the LRU path

8 years agoMore debug
Steve McIntyre [Sun, 23 Jan 2011 21:23:03 +0000 (21:23 +0000)]
More debug

8 years agoMore debug
Steve McIntyre [Sun, 23 Jan 2011 19:31:54 +0000 (19:31 +0000)]
More debug

8 years agouse debug mutex types for now
Steve McIntyre [Sun, 23 Jan 2011 17:19:42 +0000 (17:19 +0000)]
use debug mutex types for now

8 years agoadd more debug
Steve McIntyre [Sun, 23 Jan 2011 16:32:27 +0000 (16:32 +0000)]
add more debug

8 years agopthread_t is a ulong
Steve McIntyre [Sun, 23 Jan 2011 15:05:49 +0000 (15:05 +0000)]
pthread_t is a ulong

8 years agoadd thread id to debug messages
Steve McIntyre [Sun, 23 Jan 2011 15:02:07 +0000 (15:02 +0000)]
add thread id to debug messages

8 years agoFix unlikely memory leak
Steve McIntyre [Sun, 23 Jan 2011 04:14:12 +0000 (04:14 +0000)]
Fix unlikely memory leak

8 years agoStart the file worker threads detached
Steve McIntyre [Sun, 23 Jan 2011 04:13:45 +0000 (04:13 +0000)]
Start the file worker threads detached

8 years agoAdd more loggine of errors
Steve McIntyre [Sun, 23 Jan 2011 02:23:37 +0000 (02:23 +0000)]
Add more loggine of errors

8 years agoAdd more debug in cache_entry_failed
Steve McIntyre [Sun, 23 Jan 2011 02:22:35 +0000 (02:22 +0000)]
Add more debug in cache_entry_failed

8 years agoReturn errors from the encoder properly
Steve McIntyre [Sun, 23 Jan 2011 02:03:19 +0000 (02:03 +0000)]
Return errors from the encoder properly

8 years agoPass threadnum to the encoder program
Steve McIntyre [Sun, 23 Jan 2011 01:53:50 +0000 (01:53 +0000)]
Pass threadnum to the encoder program

Pass the encoder thread number through to the encoder program. Makes
it easier for a helper script to work out what to do, e.g.:

threads 0 to 3 should run on the local machine
threads 4 to 7 should run on a remote machine

8 years agoCompile with -g by default
Steve McIntyre [Sun, 23 Jan 2011 01:38:26 +0000 (01:38 +0000)]
Compile with -g by default

8 years agoMulti-threaded encoding support
Steve McIntyre [Sun, 23 Jan 2011 01:23:19 +0000 (01:23 +0000)]
Multi-threaded encoding support

Use multiple encoding threads to make the most of multiple CPUs. Lots
of changes needed, and a lot of thread work:

 * for every file we encode, track the directory it comes from. If
   we're asked to read 2 or more files in succession from the same
   directory, do "readahead". Queue up all the files in that directory
   for encoding; if we have more than one encoder thread running, then
   they'll work through the list in advance.

 * to track each of these pre-queued files properly and deal with
   cleanup, we create a thread for each one. On each encoding job,
   refcount both the original caller and the new handler thread so
   that all the paths that depend on the encoding will wait properly
   for it to finish.

 * sprinkle locks and condvars to taste to make it all work.

8 years agoDon't use cache files currently being encoded
Steve McIntyre [Sun, 23 Jan 2011 01:15:09 +0000 (01:15 +0000)]
Don't use cache files currently being encoded

If we do a lookup for a cache file to read from, if it's currently
being written to or deleted then don't use it. Ask for a re-encode
instead; if somebody else is ahead of us already encoding, we'll pick
that up fine.

8 years agoAdd compile-time control over debug logging
Steve McIntyre [Sat, 22 Jan 2011 23:32:03 +0000 (23:32 +0000)]
Add compile-time control over debug logging

8 years agoAdd compile-time control over debug logging
Steve McIntyre [Sat, 22 Jan 2011 23:27:28 +0000 (23:27 +0000)]
Add compile-time control over debug logging

8 years agoFix cache file counting at startup
Steve McIntyre [Thu, 20 Jan 2011 00:45:18 +0000 (00:45 +0000)]
Fix cache file counting at startup

8 years agoStart working on multi-threaded backend
Steve McIntyre [Wed, 19 Jan 2011 21:48:22 +0000 (21:48 +0000)]
Start working on multi-threaded backend

Add a background thread, started at mount time. Send encode requests
to this background thread instead of doing them directly.

Split out the code in encoded_size(); it now simply reports a size, or
a failure instead of triggering encoding itself. Code is cleaner this
way, and may allow for easier re-use.

For now, the background thread does the encoding directly. But it will
be much easier to have intelligence here so we can detect a sequence
of encodes and trigger encoding a complete directory in the
background.

8 years agoAdd locking to the database
Steve McIntyre [Wed, 19 Jan 2011 02:23:07 +0000 (02:23 +0000)]
Add locking to the database

Although sqlite3 is claimed to be thread-safe, my code around it
isn't! Lock all the database functions that are used at runtime, so we
don't end up freeing results in one function while trying to use them
elsewhere.

8 years agoAdd support for logging to a specified file
Steve McIntyre [Wed, 19 Jan 2011 02:09:23 +0000 (02:09 +0000)]
Add support for logging to a specified file

Will attempt to log to stderr by default, but that gets eaten by the
FUSE library code unless running with -d. Specifying --logfile <foo>
will send all the fs debug to <foo>. If used with -d as well, the FUSE
debug output will go there too.

8 years agoimprove debug: print cache states by name
Steve McIntyre [Wed, 19 Jan 2011 01:53:57 +0000 (01:53 +0000)]
improve debug: print cache states by name

8 years agolocking fixes
Steve McIntyre [Wed, 19 Jan 2011 01:51:38 +0000 (01:51 +0000)]
locking fixes

8 years agoMake statfs work properly
Steve McIntyre [Mon, 17 Jan 2011 22:04:38 +0000 (22:04 +0000)]
Make statfs work properly

Call statvfs on the basedir.
Zero the free blocks and free inodes values - we're not writeable.

8 years agoRejig command line parsing to use libfuse properly.
Steve McIntyre [Mon, 17 Jan 2011 21:58:30 +0000 (21:58 +0000)]
Rejig command line parsing to use libfuse properly.

Add command line parsing using the facilities in libfuse. Move the
config options into "struct mount_opts" as it expects.

s/format_choice/format_index/ for clarity.

8 years agoInitial re-implementation in C
Steve McIntyre [Sun, 16 Jan 2011 20:58:57 +0000 (20:58 +0000)]
Initial re-implementation in C

Uses sqlite3 backend to cache encoded file sizes.
Keep an on-disk cache to save re-encoding all the same time.

Core mount options etc. aren't quite there yet - much of it is
hard-coded. Relies on external helper scripts to do the actual
encoding yet.

8 years agoMove old perl implementation to perl subdir
Steve McIntyre [Sun, 16 Jan 2011 20:53:21 +0000 (20:53 +0000)]
Move old perl implementation to perl subdir

8 years agoAllow changing of debug level on the command line too
Steve McIntyre [Sat, 26 Jun 2010 14:21:55 +0000 (15:21 +0100)]
Allow changing of debug level on the command line too

Use the --debug option.

8 years agoTrack time along with file size for each cached file
Steve McIntyre [Sat, 26 Jun 2010 14:08:40 +0000 (15:08 +0100)]
Track time along with file size for each cached file

Store mtime as well as the size for the output, using the mtime of the
source flac file. If the source is newer than the encoded/cached file,
re-generate. Gives us a cleaner way of invalidating the cache than
simply deleting it: touch the source files.

8 years agoAdd support for encoding to a temp file
Steve McIntyre [Tue, 22 Jun 2010 15:19:26 +0000 (16:19 +0100)]
Add support for encoding to a temp file

lame doesn't seem to work properly when writing to stdout, so add
support for writing to a temporary file as well as the (better) option
of working strictly in a pipeline.

8 years agoAdd support for MP3 encoding using LAME mp3-support
Steve McIntyre [Sun, 20 Jun 2010 22:10:14 +0000 (23:10 +0100)]
Add support for MP3 encoding using LAME

Refactor encode_file() to deal with multiple encoders, both ones that
can read from flac files directly (and hence read the metadata) and
ones that can't.

Rewrite/remove a lot of code that assumed ogg support.

Split the cache for encoded file sizes into separate files (by format
and quality) so that we can safely run multiple copies of the
filesystem in parallel.

Add command line options to set output format and quality
level. Default is ogg, quality 7.

9 years agoAdd copright boilerplate
Steve McIntyre [Wed, 16 Jun 2010 20:14:38 +0000 (21:14 +0100)]
Add copright boilerplate

9 years agoInitial, mostly-working version of transcoding filesystem.
Steve McIntyre [Wed, 16 Jun 2010 20:03:57 +0000 (21:03 +0100)]
Initial, mostly-working version of transcoding filesystem.

Single-threaded only, using a cache of output file sizes to make
readdir() work faster.