Sinzui

The topics doesn't come up much, but it has so I must make a few comments. Now strictly, speaking my idea of a sandwich is turkey, lettuce, onions, on a real baguette. Most of my variations involve cheese, peppers, and another meat. Occasionally I make an exception for an Italian style sandwich on focaccia or ciabatta, a gyro, a muffuletta, and once a while a cheeseburgers.

1. chipotle mayonnaise
2. horseradish sauce
3. tzatziki sauce
4. hummus
5. tapenade
6. remoulade Sauce
7. romesco Sauce
8. Italian vinaigrette

I've been playing with Redland and its librdf recently. I'm looking for a fast replacement for Medusa's database. I'm not sure what to think. It has some new parser and query features since I played with it last year. From a storage aspect, it can do what Medusa does now, but I think I'll need more query capabilities. If I must write a query engine, there no reason it must be Medusa's so long as it works well.

I strongly feel that a GNOME metadata solution should be based on metadata standards: RDF, OWL, FOAF and use common grammars. I'm shopping for a new Medusa backend because I don't think Medusa should be in the DB business, and it needs an extensible schema. It would be nice to have a ready to go RDF DB to attach the search and indexer pieces to. I think Storage is the long term solution, but if I can get a solid db that follows stands now, I can solve scale and feature later.

The Good

• standard compliant: plays with other tools nicely
• proven: has been seen to play with others
• LGPL: is allowed to play with others
• Few dependencies: doesn't hassle others
• small: is not a burden on others
• bindings: plays with everyone

The bad

• BDB 4: will everything break when BDB 5 comes out
• Mysql: a bit of a nuisance to setup for single users
• query: applications and users need robust searching
• scalability: will this work at 100 megs, the size of my Medusa db

I've been very busy at work these past few weeks and I could not make the time to write any code. The closest thing to hacking I've done is cooking–chop chop.

1. A dynamite lemon grass chicken and summer rolls
2. an acceptable chicken korma, spinach paneer, and chholar dal.
3. A damn fine chili that my wife insisted on ruining by adding extra tomatoes.
4. Good spicy tuna rolls, mmm wasabi
5. OK Chicken fajitas. They really need a grill
6. a Fab red curry chicken.
7. A zippy Sichuan style chicken.

I cannot get enough spicy food this month. The hectic pace at work looks to be near its end. Soon I'll be putting code to disk instead of food to plates.

By prized burr coffee grind broke this morning. No coffee, no wakey. Søren Sandmann fixed GtkToolbar yesterday so I may toss the GNOME 2.4 build I've been keeping to run a few apps that don't like unstable.

I just removed my permalink to test if that is what Planet GNOME is choking on.

Five Star Stories 15 and 16 just came in the mail. This is the greatest comic/manga ever.

I've got a plan for dealing with metadata this year. It's not firm on dates, or priorities. I have a list of changes, and I'll tackle them as they seem most urgent or most beneficial. The general summary of tasks is, integrate with nautilus, make Storage with with libgda, add metadata handling to libstorage-translators, and make Storage the metadata backend.

The first things I'm doing are:

1. Make Medusa return the full set of meta data so apps like Nautilus don't need to crawl the file systems and do mime-type lookups.
2. Restore Medusa to Nautilus using using the Nautilus extension API, add simple search to the browser toolbar, and create a complex search side bar.
3. Abstract (is that verb?) the Storage DB and create a libgda facade.
4. Add metadata handling to libstorage-translators, and a means to chain them together to do consecutive processing.
5. Replace the Medusa DB with a metadata library over Storage.

I knew Mark Finlay through his emails. He was smart, insightful, and helpful. Most of all, he understood how people use computers, and he conveyed that knowledge to the GNOME community to make a great desktop. In every conversation that had an interface or usability issue, I waited to read Mark's opinions. I was surprised to learned he started at Trinity this fall, and was 'studying' computer science and learning Java. In GNOME's meritocracy, he had quickly risen to the top because he had talent way beyond his youth.

Maybe Mark excelled because he knew he had little time to leave his mark on this world. He has achieve some immortality, because his emails are archived, and his ideas are in GNOME. We had such great expectations from him.

I'll miss you Mark.

This week sucked, rotted, bl owed. What this town needs is a Chinese or Indian restaurant that I could eat my sorrows away in.
Monday Time Life, my employer, was purchased. In 60 days I'll know if I have a job. In the mean time I plan to switch to Lillian Vernon's Web architecture since there is clearly better.
Tuesday My good friend Chris, and former boss, was laid off. Anyone who knows of a company that needs a great CTO/Director of Operations/VP of E-commerce, please contact me.
Wednesday I man-handled the Web catalog data into shape when we discovered that the TV unit entered invalid data minutes before going live. Rolled out the new site in the absence of a sysadmin, only to discover the perms were bad in production.
Thursday I came in to discover the US and international sites had bad certs, The international machine was dead, the build and dev database machine was also dead.
Friday TL Staff start staking claims on my cube, my daughter stole my wallet.

A Medusa query of everything in my gnome2 build dir took 14 seconds 256 milliseconds to return 172,717 files, but display took 1 hour and 1 minute. Medusa knows everything but the icon, so msearch-gui had lookup the mime-type (again) in VFS to display the icon. Medusa could be doing a lot more, such as returning the full set of posix data, but extending it to handle icons would be difficult.

My point, in regards to Nautilus, is that it spends a lot of time traversing the disk (or net) and making repeated calls to get the specifics of each file. The file system is not designed to return a set of data like a database query, but that feature is what a modern file browser needs. That is was WinFS is reported to do. Content management systems also offer this kind of feature. Users now have gigs of data, hundreds of thousands of files. Enterprises use more than simple file systems to organize this amount of data, and so should users.

Now I question why anyone needs to see more than a few dozen files of anything; it is more than you can take in. Most likely the user is only looking for a few files. Directories are a weak means of organizing/classifying data, and users cannot easily ask for the files they want to see. A proper metadata DB would server Nautilus all the information it requests in a single lookup. A metadata DB would provide users and applications with the query features to keep the file list concise and relevant.

Directories and masses of poorly organized files is the here and now. Nautilus might ease the situation by doing a incremental display of files if a milestone is not met. It could get the list of files, sort them by access time, then begin display, so the most recent files (and commonly used) are the first the user can access.

The mime-type detection conversions on the GNOME lists is very tiring. This discussion has been focused on how to get the metadata for a file faster, but there is no issue with getting the metadata for a few dozen files. The real problems are:

1. The directory is an inefficient mechanism for organizing a large number of files.
2. We cannot query a simple file system to return a set.

I'm all for using EAs when they are available, but they are not available on all file systems that GNOME may run, hindering them as a solution. Moreover, there is a hidden cost for EAs. They take up more disk space (1-5% disk loss), apps must know to use them, and since most do not, the metadata is also stored in the file header or footer. EAs can only be accessed in the context of a single file, since the data is not organized in to anything like a database designed in the past 30 years, we cannot query them. Nor is the file system normalized to prevent duplication or orphaning of thumbnails and mime-types.

As a point of fact, Medusa
does store a table of file metadata, and I can query my file system to get a set of data like file-name, mime-type. I can query by directory, or mime-type, or keyword (emblem/topic/category), and more. This mechanism returns several thousand file matches in less than a second.

The Storage project, by it's nature, addresses the metadata problem. Though it focuses on being a smart file/data system, it can be used to manage the metadata of files outside of it. Because it is a portable file system, there are no EA issues with the OS's file system It is designed to return an arbitrary set of data matching a query like a directory, or category. I proposed a formal means of handling metadata in storage that was discussed on the storage list.

This said, I don't think Medusa or Storage is appropriate. Medusa's focus is searching, and it's underly code isn't suited to managing metadata well. Storage is as it same suggests, and it doesn't help users or applications that must use the native file system. Both Medusa and Storage provide VFS access, but neither makes it easy to read ans write metadata.

We need a layer in-between search and storage to manage metadata. It must co-opt the existing VFS methods to read and write to the metadata system when doing IO to the underlying file system. An incremental indexer is needed to collect the metadata for files not written through VFS. FAM could be used, but it will not scale; a smart indexer is need that can watch the locations that will change most. Many applications, like Web browsers, music managers, and file managers need direct access to read and write metadata without writing to the file system.

One final thought. Metadata isn't a GNOME issue, all desktops have the same issues. Freedesktop might be to right place for it. If other desktop apps like KDE were writing to the metadata DB, there would be less need for indexers.