Sauntering Coder

Smartphone as PDA

2011-03-23T20:42:00.001-07:00

After posting about potential PDAs 4 months ago, and spending several more hours researching alternatives, I finally concluded that the HTC Touch Diamond really is the perfect (or as close as the market will allow) device for me. Even better that it occasionally shows up on sale for about 30% off at Newegg, so I recently picked one up.

Several more hours of investigation have followed. I've seen it may be possible to upgrade the device to Windows Mobile 6.5, and even install Andorid. Not being ready for that kind of commitment, I've just been finding various programs and tweaks to turn it into my new PDA.

Thus begins the migration from my Palm z22 to Windows Mobile 6.1 HTC Touch Diamond

Programs

First programs I install are .NET Compact Framework 3.5, SQLServerCompact 3.5 Core, and full featured Japanese Display and Input.

After a reset, Dred Sensor, OpenMobileMaps, (Japanese) Pocket Dictionary (along with EDICT), JWPce (be sure to use the ARM "palm" version), Pocket Digital Clock, Total Commander Pocket.

System tweaks

Settings -> Personal -> Menus -- Remove Phone
Settings -> Sounds & Notifications -> Notifications
- All Phone and Messaging options -- Disable all sounds/notifications
- Event: Reminders -- Disable visible calendar popups
Settings -> Today -> Items -- Deselect TouchFLO 3D and select Pocket Digital Clock and Calendar
Settings -> Today -> Items -> Pocket Digital Clock Options -- Disable seconds display
Settings -> System -> Error Reporting -- Disable it
Settings -> Comm Manager -> Always keep Phone turned off

I hunted a long time for a (free) program to change button behavior. Dred Sensor had all the power I needed. Using Dred Sensor, I replace all the Send and End button behavior as shortcuts to Calendar and Notepad (similar to how my Palm was setup). I also set a long-press shortcut of one of the buttons to open Comm Manager for quick toggle of WiFi setting.

Copy a pic to the phone, then browse to it and select it, then set it as the background to change its translucency setting (you can't set that from Settings -> Today)

Google Maps is a very nice program, but pretty much requires constant connectivity--something I (thankfully) don't have (to pay for). So to save maps for OpenMobileMaps, first start OpenMobileMaps so it can setup its cache folders. Then download a fixed version of JTileDownloader, browse an area of interest on the OpenMaps site, download the tiles, and copy them into the cache folders on the Diamond.

I haven't yet needed to tweak the registry, but MobileRegistryEditor is my editor of choice.

Transferring Data

I've used HandyShopper on the Palm for years -- it's a brilliant program. Thankfully it's also available for Windows Mobile. Transferring the shopping list is a breeze. Just find the backed up database in %USERPROFILE%\Documents\Palm OS Desktop\<id>\Backup\HS2_Shopping.PDB and simply copy it to the device. Browse to the PDB file on the device and just click on it. Your list is now available on the Diamond.

The biggest issue I had was getting the Palm Note Pad scribbles off the z22. Since Palm all but dropped support for the Palm OS and Windows 7, I had to download their latest (and final) version of the desktop client off their website. Unfortunately that version of Palm Desktop doesn't provide an interface for Note Pad. Forum posts suggested I had to find a computer with WinXP and install the older software that came with the z22 to sync those Note Pad images off the device.

While I still have XP on my lappy, it sure seemed like a hassle -- and what about people who don't have XP anymore? So I turned to Open Source for help. After installing all variations of Palm related software in Ubuntu, nothing seemed to provide a GUI for getting those Note Pad images. Then I happened upon a man page for a command-line tool called pilot-read-notepad. To my surprise, the following command actually worked

pilot-read-notepad --type=png -p usb: > list.txt

After pressing sync on the device, I found myself with all the images exported (some of the note titles with illegal characters had to be tweaked).

Blogging with Markdown

2011-03-23T19:46:00.001-07:00

I haven't done much blogging lately, so most of the asciidoc esoteric syntax is gone from me. Meanwhile, I've been using Markdown on a couple of popular web sites. Markdown has come a long way since I first investigated a blogging markup language with code highlighting. There is now a Python module with the built-in CodeHilite extension. This post gave me the confidence that Markdown was ready for primetime.

Here's how I setup my new blogging solution with Markdown*:

Install Python 2.5+
Download Markdown source code
Download Pygments source code
Uncompress the Markdown directory
Uncompress the Pygments directory
From the Pygments directory, move the pygments and external sub-directories into the Markdown directory
Apply this patch to the Markdown source files so noclasses configuration is exposted and fenced_code is also highlighted (I'm bummed this patch isn't included in the downloaded source)
Download and install the jEdit syntax and Notepad++ syntax files

Now the following command will do the work**:

python markdown.py input.md -o html4 -x "codehilite(noclasses=True)" -x fenced_code -f output.html

I still have to manually create the blog post in Blogger, but then it's just a copy-paste of the html, and I'm done.

_{* I don't like installing Python modules because I rarely use them--hence the simple uncompressing and use in-place.}

_{** Like with my previous blogging solution, I like the syntax CSS to be inline.}

Using the iPod Touch as a PDA

2010-11-04T21:16:00.000-07:00

I've been sporting the stylish Palm z22 for several years now (originally purchased for $100), and its battery is beginning to fail. Would it be too much to ask for something similar at a similar price point? Apparently it is. There is only one company that still makes normal PDAs: HP with their iPaq 111 (3 times the cost!) and 211. So if I can't get a PDA, it seems Portable Media Devices (PMD) are the only alternative.

The iPod Touch almost meets the bill, but still lacks a way to properly scribble notes. Researching compatible styluses led to the BoxWave Mini stylus, which is adequate, but falls short of my 5 year old PDA's accuracy. I could settle with that, if only there was a decent case that would hold it for me. Again, I must be asking way too much because there are currently only two* (2) on the market, and they are ugly as sin.

Fine, I won't get the iPod Touch. So what's left? The Creative Zen seemed to be able to sync with MS Outlook...about 4 years ago.

If I can't get a PDA, or a PMD, I guess I could just get a stinkin' smartphone and just never activate it. Lo and behold that's not such a bad idea. The HTC Tilt and HTC Touch Diamond are both available unlocked on Newegg for not terrible prices. Plus they even have a camera that's 3x better than my cellphone's. Just wish I could try out those devices before committing $half-a-PS3 to it.

_{* Got my links messed up. There is one case that I really like, but it's only compatible with iPhone.}

Parentheses

2010-01-27T21:40:00.001-08:00

The fellows over at the newLISP fan club forums are a witty bunch. During yet another discussion in the long history of "how might all those parentheses be removed from Lisp" discussions, Elica solved this age old conundrum ever so gracefully.

There is one easy recursive solution, but I’m afraid you will not like it, although you may enjoy it. Here it goes:

To solve the problem with parentheses, we can replace them by square brackets. However, this will introduce a new problem - a problem with the square brackets. To solve it, we can replace brackets with parentheses, which will reraise the initial problem. Fortunately, we already know how to solve it.

So, everything is OK - the initial problem is reduced to a sequence of problems, each of which can be solved trivially.

Java and VNC on Windows

2010-01-27T20:04:00.001-08:00

Sometime between JRE 1.6.0 update 3 and 1.6.0 update 15, something changed.

I use a VNC program heavily, and was surprised to see all Java Swing GUIs become almost useless because they would never refresh through VNC. Yet at the same time Netbeans still worked fine over VNC.

I sought high and low, but what kind of search string would you use for this issue?

As a work around at the time, I set the JAVA_HOME to an older JRE, and updated some of my Java program shortcuts to use "C:\Program Files\Java\jre1.6.0_3\bin\javaw.exe"

Today I finally found the issue and fix.

J2D_D3D=false

This problem might occur with any other kind of Remote Desktop program, or perhaps it is just those that use a special video driver.

Oh and Netbeans seems to run under the latest JDK installed instead of the latest JRE. Guess I should update that sometime.

Aion Beta: Closed vs. Open

2009-09-08T22:37:00.001-07:00

Q. What is the difference between all the previous Aion closed betas and the current Aion open beta?: A. The “open” beta also includes some lucky FilePlanet paid subscribers that got a limited “open” beta key.

Mozilla Store H4x3d

2009-08-05T20:31:00.000-07:00

Over a year ago I ordered the satanic Firefox t-shirt from the Mozilla store. I really enjoy using Firefox on a daily basis, and can certainly stand behind it via t-shrit. Unfortunately the Mozilla Store experience failed to be on par with its advertised web browser experience.

Once I got the shirt in the e-kart, I had to create a new registration. Sure, registering for yet another web site is a pain (almost 50 different sites now have my credit card into, and each has their own username/password), but such is the internet today. However, after signing up, I was not re-directed to the checkout page. I assumed I needed to select the item again, so I annoyingly went back through the product listing and re-added it. This finally led me back to my shopping-cart, where I then found it contained 2 of the same item.

Removing the duplicate, I continued on and made my purchase. I then wanted to be secure by logging out, but after several minutes of searching, I was unable to find any link in which to log out. Went ahead and closed the browser, and assumed that was the last I needed to worry.

Later that day I receive an email.

Before we can release the order to ship, we need to verify the billing and shipping information. We have ran the credit card and show that parts of the billing information supplied to this order does not match the billing information for this credit card.

???

I’ve made several dozen online orders in my day, and I’ve never had a problem with my credit card. I checked the order confirmation email, and logged back onto the site. Both places had perfectly correct addresses.

Address line 1: 123 Main St. #555
Address line 2: <blank>

Even though this has never been a problem, I went ahead and added my middle initial to my name and put my apartment number on the address 2nd line. After confirming my updated info, I was dismayed to find my address displayed like this:

Address line 1: 123 Main St.
Address line 2: #555123 Main St.

!!!

I’m not particularly paranoid about security, but after all this, I hardly felt confident sending all my address information over unencrypted email. I replied that I double-checked my address info, and everything was correct. If they wanted me to re-send it anyway, I would appreciate additional details on why.

The patient fellow explained in his reply, "if you are a new customer ordering then, for your safety and ours the order gets held for verification purpose." Well that’s nice, but it still didn’t make much sense. He included my address in his email, which I added a few details that I didn’t think were necessary (middle initial and name of company at the shipping address). I eventually got my shirt, and have worn it ever since.

After all this, I can’t say I’m very surprised with the news that the Mozilla store was hacked. If my ordering experience is any indication of how the store’s backend was handled, then they were just begging for problems.

A Tale of Two TVs...+2

2009-07-27T05:21:00.001-07:00

I’ve owned a cheap, 20", super ultra generic Best Buy TV for the last 5 years. It has been wonderfully reliable and well used. I think my desire for something bigger first started when I played Gears of War at a friend’s house on a large screen. The display and experience was truly awesome. Since then I’ve also felt my little screen didn’t do justice to The Legend of Zelda: Twilight Princess. Old news to anyone in the know, but I was crushed to learn the Wii doesn’t output anything HD. It does do wide screen at lest, so a new TV wouldn’t be completely wasted on it (and good luck finding a 480p wide screen).

It wasn’t until I had researched 2 months, purchased 4 TVs, then returned 3 of them, that all my requirements had been flushed out.

32" wide-screen LCD
Closed Captioning on mute
The closed captioning starts appearing shortly after pressing mute
Passes The Legend of Zelda: Twilight Princess display calibration screen
Minimal input-to-display lag
Easily returnable

The 4th TV finally met my needs (almost). Given how hard it is to find any worthwhile information on TVs (and all electronics in general) on the internet, thought I’d share some details.

Consumer Reports

I’ve been an off-and-on registered member of Consumer Reports. Unlike fickle forums, their information is objective and substantiated. Unfortunately their TVs coverage is pretty lacking (but given the number of TVs on the market, it’s unlikely they could really keep up). If your TV of interest isn’t reviewed on the site, then the only valuable information they offer is a general brand reliability rating. Their surveys give a glimpse of what percentage of TVs are defective for a particular brand. The numbers seem to support the idea that brands do maintain a consistent level of reliability.

Vizio 32" VO320E — $400 (normally $450)

As a newcomer to the big screen TV market, I was surprised that the otherwise unheard of Vizio brand is a major player. Consumer Reports rate their reliability as one of the best. I hear they also have a pretty hefty market share. This TV on sale at Walmart has an excellent cost-to-quality ratio.

Since it was my first TV, it wasn’t until after I purchased it that I realized I really like how my old CRT could show Closed Captioning when muted. This isn’t a feature most people even realize exists, and it is never mentioned on retail sites. You have to go to the manufacturer’s site, download the product’s manual and read all the details. In any case, I strongly recommend people do that for any TV they plan to buy.

The only other down side of this TV is the simplistic remote. There are no shortcuts to menu options on the remote. For example, if you want to change the “zoom” display of a channel, it takes 5-6 button presses to traverse the menu system and change the settings.

Insignia (Best Buy generic) 32" NS-L32Q-10A — $400

After additional research, I narrowed it down to this Insignia and the RCA below. When I learned from an employee that Insignia (a generic Best Buy brand) is at least partially made by LG, I went for this one. Consumer Reports says LG reliability isn’t one of the best, but at least it’s on the list (unlike RCA).

The picture quality of this TV looks quite nice, especially when compared next to the RCA, and it has my coveted Closed Captioning on mute.

Sadly it was lacking in several other ways. The signal to some of my cable channels is a bit weak, so the picture is a little fuzzy on my CRT. On the Insignia, this weak signal sometimes exhibits itself as no audio. I found the sound could be acquired by turning the TV off, then back on. Unfortunately the sound goes away again if you change to another channel and return. This didn’t always happen, but it was enough to be very troubling. In general, this TV doesn’t handle corrupted video signal very well.

Overall the TV is just very slow. It takes about 10 to 15 seconds to turn on, and channel changing often takes like 3 to 4 seconds.

While all this is pretty lame, it was still usable—that is until I tried to play video games on it. It has a seriously bad lag from receiving input to displaying the image.

I don’t think there’s any official name for this “lag” value, and it’s never reported by TV manufacturers. Not all hope is lost though. There does seem to be some brand consistency. Looking around Best Buy, I noticed most, if not all the playable video game demos were setup on Samsung LCDs. I suspect Sonys are generally pretty good with lag as well. You can also find a sort of “game mode” feature on various TVs that skips all the image processing and just blasts the video to the screen as quickly as possible.

RCA 32" L32HD41 — $350 (normally $400)

According to the specs, this TV has a higher contrast ratio than the Insignia, but it sure didn’t look like that when sitting side-by-side. The picture was very dark and bland, and never could get bright enough, or with enough contrast to pass the Zelda Twilight Princess calibration screen.

Everything else about the TV is fine. Its input-to-display lag was pretty good, and it’s the quickest to display CC after the mute button was pressed.

Sony - BRAVIA 32" KDL-32L5000 — $500 (normally $550)

After three <= $400 TVs, it was pretty clear I can’t be cheap if I want a TV to really meet my needs. It was either this Sony or a Toshiba of the same price. I found a mention of Playstation 3 in this TV’s manual (on the Sony site). I hoped this meant its lag was small enough for video games. Sure enough, its input-to-display lag is very usable.

The picture passes Zelda Twilight Princess screen calibration without problems. All-in-all it’s quite a nice TV. The only thing about it that bugs me is the 30 seconds it takes for Closed Captioning to start showing after mute is pressed. That delay is also reset if you press pretty much any button on the remote. Makes me wish there was more interest in TV firmware hacking out there.

Final notes

I have yet to find a Samsung that does CC on mute.

I’ve now dropped about $100 in new cables (all ordered online to avoid highway robbery) to make my home theater system complete. So this whole purchase has set met back over $600. I will also be paying higher energy costs for this new TV vs. my old CRT. That little guy will be missed.

Compounding Hibernate Problems 2: The Enterprise Solution Strikes Back

2009-06-26T19:17:00.000-07:00

It is a dark time for the
developers. Although Hibernate
can handle simple schema designs,
existing databases have driven the
coders from their lighthearted
news aggregation sites to
pursue answers from across
the internet.

Again I’m trying to add some table relations with Hibernate’s Enterprise handling of composite primary keys getting in the way.

Disclaimer: This is a somewhat contrived example, but it accurately reflects the problems with a very real database.

Back with good ol' Derby.

jdbc:derby://localhost:1527/MyDB;create=true

username=myschema
password=whatever

ddl

create table vehicle (
    make_id integer,
    model_id integer,
    name varchar(50),
    PRIMARY KEY (make_id, model_id)
);

create table maker_country (
    make_id integer,
    country_id integer,
    make_country varchar(50),
    PRIMARY KEY (make_id, country_id)
);


insert into vehicle values (1, 0, 'Chevrolet Corvette');
insert into vehicle values (1, 1, 'Chevrolet Malibu');
insert into vehicle values (2, 0, 'Ford Focus');
insert into vehicle values (2, 1, 'Ford Mustang');
insert into vehicle values (3, 0, 'Honda Accord');
insert into vehicle values (3, 1, 'Honda Odyssey');
insert into vehicle values (4, 0, 'Toyota Avalon');
insert into vehicle values (4, 1, 'Toyota Camry');
insert into vehicle values (5, 1, 'Volkswagen Bug');

insert into maker_country values (1, 1, 'Chevrolet makes cars for USA');
insert into maker_country values (2, 1, 'Ford makes cars for USA');
insert into maker_country values (3, 2, 'Honda makes cars for Japan');
insert into maker_country values (3, 1, 'Honda makes cars for USA');
insert into maker_country values (4, 2, 'Toyota makes cars for Japan');
insert into maker_country values (4, 1, 'Toyota makes cars for USA');
insert into maker_country values (5, 3, 'Volkswagen makes cars for Germany');
insert into maker_country values (5, 1, 'Volkswagen makes cars for USA');

.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC
        "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
        "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
    <session-factory>
        <property name="hibernate.connection.driver_class">org.apache.derby.jdbc.EmbeddedDriver</property>
        <property name="hibernate.dialect">org.hibernate.dialect.DerbyDialect</property>
        <property name="hibernate.connection.url">jdbc:derby://localhost:1527/MyDB;create=true</property>
        <property name="hibernate.connection.username">myschema</property>
        <property name="hibernate.connection.password">whatever</property>
        <property name="hibernate.default_schema">MYSCHEMA</property>
        <property name="hibernate.current_session_context_class">thread</property>
        <property name="hibernate.show_sql">true</property>
    </session-factory>
</hibernate-configuration>

Of course we know this won’t work…

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-reverse-engineering PUBLIC
    "-//Hibernate/Hibernate Reverse Engineering DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-reverse-engineering-3.0.dtd" >

<hibernate-reverse-engineering>
    <table-filter match-name="MAKER_COUNTRY"/>
    <table-filter match-name="VEHICLE"/>

    <table name="VEHICLE">
        <foreign-key foreign-table="MAKER_COUNTRY">
            <column-ref local-column="MAKE_ID" foreign-column="MAKE_ID" />
        </foreign-key>
    </table>

</hibernate-reverse-engineering>

Foreign key (FK3F2F1B0C64FEF3DE:VEHICLE [MAKE_ID])) must have same number of columns as the referenced primary key (MAKER_COUNTRY [MAKE_ID,COUNTRY_ID])

So we stick with the default generated reverse engineering. We can use it to generate only the mapping files (no .java classes yet).

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-reverse-engineering PUBLIC
    "-//Hibernate/Hibernate Reverse Engineering DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-reverse-engineering-3.0.dtd" >

<hibernate-reverse-engineering>
    <table-filter match-name="MAKER_COUNTRY"/>
    <table-filter match-name="VEHICLE"/>
</hibernate-reverse-engineering>

Then we can edit the mapping file to add a relationship between two tables.

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="Vehicle" table="VEHICLE">
        <composite-id name="id" class="VehicleId">
            <key-property name="makeId" type="int">
                <column name="MAKE_ID" />
            </key-property>
            <key-property name="modelId" type="int">
                <column name="MODEL_ID" />
            </key-property>
        </composite-id>
        <property name="name" type="string">
            <column name="NAME" length="50" />
        </property>

        <!-- Added this: -->
        <set name="makerCountries" inverse="false" lazy="true">
            <key property-ref="makeId" column="MAKE_ID" />
            <one-to-many class="MakerCountry" />
        </set>

    </class>
</hibernate-mapping>

No problem, right?

org.hibernate.MappingException: property-ref [makeId] not found on entity [Vehicle]

Ok, so it can’t see its own property. What if reference its key’s property.

        ...
            <key property-ref="id.makeId" column="MAKE_ID" />
        ...

It generated some classes. Let’s give it a run.

Test.java

import java.net.URL;
import java.util.List;
import java.util.Set;
import org.hibernate.Query;
import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.hibernate.cfg.AnnotationConfiguration;

public class Test {
    static SessionFactory sessionFactory;
    static Session session;
    public static void main(String[] args) {
        URL url = Test.class.getResource("/hibernate.cfg.xml");
        sessionFactory = new AnnotationConfiguration().configure(url).buildSessionFactory();
        session = sessionFactory.getCurrentSession();
        org.hibernate.Transaction tx = session.beginTransaction();

        Query q = session.createQuery("from Vehicle as v where v.id.makeId = 1");
        List<Vehicle> lst = null;
        lst = q.list();
        if (lst.size() > 0) {
            Vehicle v = lst.get(0);
            System.out.println(v.getName());
            Set<MakerCountry> makerCountries = v.getMakerCountries();
            System.out.println(makerCountries.size());
        }
    }
}

Nope.

Hibernate: select vehicle0_.MAKE_ID as MAKE1_1_, vehicle0_.MODEL_ID as MODEL2_1_, vehicle0_.NAME as NAME1_ from MYSCHEMA.VEHICLE vehicle0_ where vehicle0_.MAKE_ID=1
Exception in thread "main" org.hibernate.HibernateException: Unable to resolve property: id.makeId

Fine, we’ll add the property ourselves.

        ...
        <property name="makeId" type="int" insert="false" update="false">
            <column name="MAKE_ID" />
        </property>
        <set name="makerCountries" inverse="false" lazy="true">
            <key property-ref="makeId" column="MAKE_ID" />
            <one-to-many class="MakerCountry" />
        </set>
        ...

Now umm…

Hibernate: select vehicle0_.MAKE_ID as MAKE1_1_, vehicle0_.MODEL_ID as MODEL2_1_, vehicle0_.NAME as NAME1_ from MYSCHEMA.VEHICLE vehicle0_ where vehicle0_.MAKE_ID=1
Chevrolet Corvette
Exception in thread "main" org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: Vehicle.makerCountries, no session or session was closed

"no session or session was closed"?? What does that mean?

Whatever, we’ll set lazy to false.

        ...
        <set name="makerCountries" inverse="false" lazy="false">
        ...

and…

Hibernate: select vehicle0_.MAKE_ID as MAKE1_1_, vehicle0_.MODEL_ID as MODEL2_1_, vehicle0_.NAME as NAME1_ from MYSCHEMA.VEHICLE vehicle0_ where vehicle0_.MAKE_ID=1
Exception in thread "main" java.lang.ClassCastException: java.lang.Integer

I think Luke Skywalker summed it up quite poignantly when he said

NOOOOOOOOOOOOOO!!

Lazy Hibernate

2009-06-23T20:15:00.000-07:00

Let’s play with large collections.

ddl

CREATE TABLE popular_band (
    band_id AS INTEGER PRIMARY KEY,
    band_name AS VARCHAR
);

CREATE TABLE fan (
    fan_id AS INTEGER PRIMARY KEY,
    fan_name AS VARCHAR,
    favorite_band_id AS INTEGER
);

ALTER TABLE fan ADD fans_fk_popular_band FOREIGN KEY (favorite_band_id)
REFERENCES popular_band(band_id);

INSERT INTO popular_band VALUES (1, "Beatles");

INSERT INTO fan VALUES (1, "Jim", 1);
INSERT INTO fan VALUES (2, "Bob", 1);
INSERT INTO fan VALUES (3, "Sue", 1);
INSERT INTO fan VALUES (4, "Mary", 1);
-- 100,000 more ...
INSERT INTO fan VALUES (100005, "Alex", 1);

After reverse engineering this schema with Hibernate, your mappings should look something like this.

PopularBand.hbm.xml

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="PopularBand" table="POPULAR_BAND">
        <id name="bandId" type="long">
            <column name="BAND_ID" />
        </id>
        <property name="bandName" type="string">
            <column name="BAND_NAME" />
        </property>
        <set name="fans" inverse="true" lazy="true" table="FAN" fetch="select">
            <key>
                <column name="FAVORITE_BAND_ID" precision="10" scale="0" />
            </key>
            <one-to-many class="Fan" />
        </set>
    </class>

    <class name="Fan" table="FAN">
        <id name="fanId" type="long">
            <column name="FAN_ID" />
        </id>
        <property name="fanName" type="string">
            <column name="FAN_NAME" />
        </property>
        <many-to-one name="favoriteBand" class="PopularBand" fetch="select">
            <column name="FAVORITE_BAND_ID" />
        </many-to-one>
    </class>
</hibernate-mapping>

The Beatles, being a very popular band, have many fans. I just want to look at 10, in no particular order. That shouldn’t be hard, right?

public static void print10fans(PopularBand pb) {
    Set<Fan> fans = pb.getFans();
    Iterator<Fan> iter = fans.iterator();
    for (int i=0; i < 10 && iter.hasNext(); i++) {
        System.out.println(iter.next().getFanName());
    }
}

And the output is

java.lang.OutOfMemoryError: Java heap space
...

Oops, our very large collection is very large. Maybe the manual can help us.

lazy (optional - defaults to true) may be used to disable lazy fetching and specify that the association is always eagerly fetched, or to enable "extra-lazy" fetching where most operations do not initialize the collection (suitable for very large collections)

Perfect! Just a quick change

<set name="fans" inverse="true" lazy="extra" table="FAN" fetch="select">
                                      ^^^^^

and we can try again.

java.lang.OutOfMemoryError: Java heap space
...

Ok, just what does this "extra-lazy" really mean anyway? The manual is very silent on such details, so we turn to other sources (emphasis added).

Extra lazy doesn’t do much on sets. It is actually quite useful only on the size method which triggers a select count on the database. However, iterating over a set will instantiate the whole collection. On maps and lists, extra lazy also enables fetching an entry by index. Still iterating over a list or a map will fetch the whole collection from database.

So when the manual says

…most operations do not initialize the collection (suitable for very large collections)

it actually means

…only the size() method does not initialize the collection, unless it’s a map or list, then you can also fetch by index (but otherwise your very large collection is destined to crash your program)

You can use "Filtering collections" to specify how many results you want, but doing that seems to turn the whole idea of “Plain-Old-Java-Objects” into “Plain-Old-Java-Objects-Unless-Your-Collections-Are-Too-Big-Then-You-Have-To-Use-Framework-Specific-API”

Fastest bulk import into sqlite

2009-05-11T12:50:00.000-07:00

I just wanted to populate a sqlite database with 100 million rows of data from a 1.8 gigabyte tab delimited file (tsv).

commands.txt

.echo ON

.read create_table_without_pk.sql

PRAGMA cache_size = 400000;
PRAGMA synchronous = OFF;
PRAGMA journal_mode = OFF;
PRAGMA locking_mode = EXCLUSIVE;
PRAGMA count_changes = OFF;
PRAGMA temp_store = MEMORY;
PRAGMA auto_vacuum = NONE;

.separator "\t"
.import a_tab_seprated_table.txt mytable

BEGIN;
.read add_indexes.sql
COMMIT;

.exit

PRAGMA cache_size = 400000 lets sqlite allocate about 460MB of RAM to work with. You can find more detailed explaination of these PRAGMA options on the Sqlite site.

sqlite3 mydb.db < commands.txt

Using this approach, the inserting of data took only about 20 mins.

However, adding a composite primary key (i.e. unique index) and a non-unique composite index ran for 70+ hours before I just canceled it.

If you can pre-sort your data in the order it is indexed, the import speed is practically the same whether you create the index before or after the import. However, if you create the index before the import, the database data and indexes will be intermingled with each other, thus fragmented (but this won’t affect the file size). The fragmentation can be corrected by running the VACUUM command.

There is a powerful ExternalSort implementation as part of the Java Small Text library. Look under Documentation→Small Text Utils tutorial for info about it.

Compounding Hibernate Problems

2009-05-01T13:21:00.000-07:00

Platform information

Eclipse Platform
Version: 3.4.2
Build id: M20090211-1700

Hibernate Tools 3.2.4.GA-R200903141626-H5
and
Hibernate Tools 3.2.4.GA-N200904280907-H52

Windows XP Pro SP3

JRE 6

Tested on Oracle 10g and Apache Derby

Two problems with Hibernate Tools reverse engineering:

Unable to create a relationship from a non-primary key column in one table to one column of a composite primary key in another table, and visa-versa.
Creating a relationship from a column in a composite primary key of one table to a non-primary key column in another table produces incorrect mapping file (I have submitted a poorly written bug report for this one).

These issues will be explorered in the following example. For simplicity we’ll use an Apache Derby database.

Start the database, and connect to it.

jdbc:derby://localhost:1527/myDB;create=true;user=me;password=mine

Create these tables in the myDB database, in the ME schema.

ddl

create table vehicles (
    make_id integer,
    model_id integer,
    name varchar(50),
    PRIMARY KEY (make_id, model_id)
);

create table owners (
    owner_id integer PRIMARY KEY,
    favorite_make_id integer,
    name varchar(20)
);

insert into vehicles values (1, 0, 'Chevrolet Corvette');
insert into vehicles values (1, 1, 'Chevrolet Malibu');
insert into vehicles values (2, 0, 'Ford Focus');
insert into vehicles values (2, 1, 'Ford Mustang');
insert into vehicles values (3, 0, 'Honda Accord');
insert into vehicles values (3, 1, 'Honda Odyssey');
insert into vehicles values (4, 0, 'Toyota Avalon');
insert into vehicles values (4, 1, 'Toyota Camry');

insert into owners values (50, 3, 'Jim');
insert into owners values (51, 1, 'Alex');
insert into owners values (52, 4, 'Sue');
insert into owners values (53, 1, 'Casey');
insert into owners values (54, 3, 'Ted');
insert into owners values (55, 2, 'Mary');

Create this Hibernate Configuration file.

.cfg.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC
        "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
        "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
    <session-factory>
        <property name="hibernate.connection.driver_class">org.apache.derby.jdbc.EmbeddedDriver</property>
        <property name="hibernate.connection.url">jdbc:derby://localhost:1527/myDB</property>
        <property name="hibernate.connection.username">me</property>
        <property name="hibernate.connection.password">mine</property>
        <property name="hibernate.dialect">org.hibernate.dialect.DerbyDialect</property>
        <property name="hibernate.current_session_context_class">thread</property>
        <property name="hibernate.default_schema">ME</property>
        <property name="hibernate.show_sql">true</property>
    </session-factory>
</hibernate-configuration>

Setup a Hibernate Console Configuration with the configuration above and connection to the Derby database.

Create this Hibernate reverse engineering configuration.

.reveng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-reverse-engineering PUBLIC
    "-//Hibernate/Hibernate Reverse Engineering DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-reverse-engineering-3.0.dtd" >

<hibernate-reverse-engineering>
    <table-filter match-name="OWNERS"/>
    <table-filter match-name="VEHICLES"/>

    <table name="OWNERS">
        <foreign-key foreign-table="VEHICLES">
            <column-ref local-column="FAVORITE_MAKE_ID" foreign-column="MAKE_ID" />
        </foreign-key>
    </table>

</hibernate-reverse-engineering>

Setup a Hibernate Code Generation configuration like this:
[Main tab] [Exporters tab]

Trying to run it produces this error:

org.hibernate.MappingException: Foreign key (FK8BBD58205C954C50:OWNERS [FAVORITE_MAKE_ID])) must have same number of columns as the referenced primary key (VEHICLES [MAKE_ID,MODEL_ID])
Foreign key (FK8BBD58205C954C50:OWNERS [FAVORITE_MAKE_ID])) must have same number of columns as the referenced primary key (VEHICLES [MAKE_ID,MODEL_ID])

Searching for "must have same number of columns as the referenced primary key" on Google returns 6 pages of results. You can find references to this error as early as 2005.

The Other Way (incorrect)

So let’s try the other way, even though it’s incorrect.

Change the Hibernate reverse engineering configuration to this.

.reveng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-reverse-engineering PUBLIC
    "-//Hibernate/Hibernate Reverse Engineering DTD 3.0//EN"
    "http://hibernate.sourceforge.net/hibernate-reverse-engineering-3.0.dtd" >

<hibernate-reverse-engineering>
    <table-filter match-name="OWNERS"/>
    <table-filter match-name="VEHICLES"/>

    <table name="VEHICLES">
        <foreign-key foreign-table="OWNERS">
            <column-ref local-column="MAKE_ID" foreign-column="FAVORITE_MAKE_ID" />
        </foreign-key>
    </table>

</hibernate-reverse-engineering>

Now the Code Generation configuration will run, but it produces an incorrect mapping file.

Vehicles.hbm.xml

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="Vehicles" table="VEHICLES">
        <composite-id name="id" class="VehiclesId">
            <key-property name="makeId" type="int">
                <column name="MAKE_ID" />
            </key-property>
            <key-property name="modelId" type="int">
                <column name="MODEL_ID" />
            </key-property>
        </composite-id>
        <many-to-one name="owners" class="Owners" update="false" insert="false" fetch="select">
            <column name="MAKE_ID" not-null="true" />
        </many-to-one>
        <property name="name" type="string">
            <column name="NAME" length="50" />
        </property>
    </class>
</hibernate-mapping>

The many-to-one tag should have this additional attribute:

property-ref="favoriteMakeId"

Because it doesn’t, this code will fail:

Main.java

import org.hibernate.cfg.AnnotationConfiguration;
import org.hibernate.*;

public class Main {

    public static void main(String[] args) {
        System.out.println("Connecting to Derby database.");
        SessionFactory sf = new AnnotationConfiguration().configure().buildSessionFactory();
        Session session = sf.getCurrentSession();

        System.out.println("Querying for some vehciles.");
        session.beginTransaction();
        Query q = session.createQuery("from Vehicles as v where v.id.modelId = 0");
        java.util.List<Vehicles> vehicleList = q.list();

        System.out.println("Got " + vehicleList.size() + " matches.");

        for (Vehicles vehicle : vehicleList) {

            System.out.println("Getting an owner that likes the maker of " + vehicle.getName());
            Owners owner = vehicle.getOwners();
            try {
                System.out.println(owner.getName() + " likes it.");
            } catch (ObjectNotFoundException ex) {
                ex.printStackTrace();
                System.err.println("This fails because it was matching VEHCILES.MAKE_ID to OWNERS.OWNER_ID");
                System.err.println("when it SHOULD have matched to OWNERS.FAVORITE_MAKE_ID like the reveng says.");
            }
        }
    }

}

Program output:

Connecting to Derby database.
Querying for some vehciles.
Hibernate: select vehicles0_.MAKE_ID as MAKE1_0_, vehicles0_.MODEL_ID as MODEL2_0_, vehicles0_.NAME as NAME0_ from ME.VEHICLES vehicles0_ where vehicles0_.MODEL_ID=0
Got 4 matches.
Getting an owner that likes the maker of Chevrolet Corvette
Hibernate: select owners0_.OWNER_ID as OWNER1_1_0_, owners0_.FAVORITE_MAKE_ID as FAVORITE2_1_0_, owners0_.NAME as NAME1_0_ from ME.OWNERS owners0_ where owners0_.OWNER_ID=?
org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [Owners#1]
    at org.hibernate.impl.SessionFactoryImpl$2.handleEntityNotFound(SessionFactoryImpl.java:409)
    at org.hibernate.proxy.AbstractLazyInitializer.checkTargetState(AbstractLazyInitializer.java:108)
    at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:97)
    at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:140)
    at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:190)
    at Owners_$$_javassist_1.getName(Owners_$$_javassist_1.java)
    at Main.main(Main.java:23)
This fails because it was matching VEHCILES.MAKE_ID to OWNERS.OWNER_ID
...

However, even if you manually add the property-ref, when you run it with the example program, it returns an error of “More than one row with the given identifier was found”.

Connecting to Derby database.
Querying for some vehciles.
Hibernate: select vehicles0_.MAKE_ID as MAKE1_0_, vehicles0_.MODEL_ID as MODEL2_0_, vehicles0_.NAME as NAME0_ from ME.VEHICLES vehicles0_ where vehicles0_.MODEL_ID=0
Hibernate: select owners0_.OWNER_ID as OWNER1_1_0_, owners0_.FAVORITE_MAKE_ID as FAVORITE2_1_0_, owners0_.NAME as NAME1_0_ from ME.OWNERS owners0_ where owners0_.FAVORITE_MAKE_ID=?
Exception in thread "main" org.hibernate.HibernateException: More than one row with the given identifier was found: 1, for class: Owners
    at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:92)
    at org.hibernate.loader.entity.EntityLoader.loadByUniqueKey(EntityLoader.java:108)
    at org.hibernate.persister.entity.AbstractEntityPersister.loadByUniqueKey(AbstractEntityPersister.java:1672)
    at org.hibernate.type.EntityType.loadByUniqueKey(EntityType.java:641)
    at org.hibernate.type.EntityType.resolve(EntityType.java:415)
    at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:139)
    at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:877)
    at org.hibernate.loader.Loader.doQuery(Loader.java:752)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
    at org.hibernate.loader.Loader.doList(Loader.java:2228)
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
    at org.hibernate.loader.Loader.list(Loader.java:2120)
    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
    at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361)
    at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
    at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148)
    at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
    at Main.main(Main.java:14)

This is because the example column has non-unique entries. If it only had unique entires, our manual property-ref fix would work.

Doing it manually

Hibernate reverse engineering has pretty much no support for special table relationships (e.g. a non-inverse one-to-many). The only real solution to this problem is to manually create the mapping files and classes yourself.

Some links that probably won’t help at all:

Trying to use a composite-id with key-many-to-one won’t work because the foreign column needs to be a primary key.

Here is how the mapping of Vehicles → Owners should be:

Vehicles.hbm.xml

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="Vehicles" table="VEHICLES">
        <composite-id mapped="false" >
            <key-property name="makeId" type="int" >
                <column name="MAKE_ID" />
            </key-property>
            <key-property name="modelId" type="int">
                <column name="MODEL_ID" />
            </key-property>
        </composite-id>
        <property name="name" type="string">
            <column name="NAME" length="50" />
        </property>
        <bag name="owners">
            <key>
                <column name="FAVORITE_MAKE_ID" />
            </key>
            <one-to-many class="Owners" />
        </bag>
    </class>
</hibernate-mapping>

Some problems here:

Any kind of collection mapping (set, list, bag, array, map) require a key element be defined.
The key generates an error because it doesn’t map the “same number of columns” (i.e. it needs 2 columns to match the 2 column primary key). It’s saying it needs another key defined.
So why not just get rid of the primary key? Because the class element requires a id or composite-id element. And we can’t make just MAKE_ID or MODEL_ID the primary key because those won’t be unique.

Solution

At long last! For Owners → Vehicles, this actually works.

Owners.hbm.xml

<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
    <class name="Owners" table="OWNERS">
        <id name="ownerId" type="int">
            <column name="OWNER_ID" />
            <generator class="assigned" />
        </id>
        <property name="favoriteMakeId" type="int">
            <column name="FAVORITE_MAKE_ID" />
        </property>
        <property name="name" type="string">
            <column name="NAME" length="20" />
        </property>
        <set name="vehicles" inverse="false">
            <key property-ref="favoriteMakeId" column="MAKE_ID" />
            <one-to-many class="Vehicles" />
        </set>
      </class>
</hibernate-mapping>

There still doesn’t appear to be any method to properly associate Vehicles → Owners, but thankfully the mapping file above is all I really need for my case.

There also doesn’t appear to be any way to produce a mapping file like this using Hibernate Tools reverse engineering, so I just have to make it manually.

Going back through some of the many links I visited while searching for help, I noticed this page also basically explains the solution above.

How to build Hibernate Core...I think?

2009-04-28T10:11:00.001-07:00

Checkout the lastest SVN trunk from http://anonsvn.jboss.org/repos/hibernate/core/trunk.
I got rev 16450

Set your JAVA_HOME environment variable to a Java 5 JDK (it fails with Java 6).
Mine is set to C:\Program Files\Java\jdk1.5.0_17

Edit parent/pom.xml to add these lines in bold.

...
         </mailingList>
     </mailingLists>


    <repositories>
        <repository>
            <id>jboss</id>
            <url>http://repository.jboss.com/maven2/</url>
        </repository>

        <repository>
            <id>jboss-snapshot</id>
            <url>http://snapshots.jboss.org/maven2</url>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
            <releases>
                <enabled>false</enabled>
            </releases>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>jboss</id>
            <url>http://repository.jboss.com/maven2/</url>
        </pluginRepository>
    </pluginRepositories>


     <build>
         <plugins>
             <plugin>
...

Download Maven.
I got apache-maven-2.1.0-bin.zip

Unpack Maven somewhere, cd to the hibernate-core trunk directory and run this command.

<path to apache-maven>/bin/mvn package

This should automatically download everything, build the whole thing, and run the tests.

My test results were

Tests run: 117, Failures: 0, Errors: 56, Skipped: 0

At this point I have no idea what to do…

Highlighting Code Snippets

2009-04-24T12:55:00.001-07:00

Windows Live Writer seems to be a popular blogging client, especially with the Code Formatting plugin. After the last post however, its shortcomings became very apparent. The writer’s formatting is fickle and limited. The code formatter is messy and unreliable (it refused to highlight the code in the last post).

It’s sad that with all our technology, we still have yet to make very reliable WYSIWYG text editors. That means writing HTML directly, or finding an intermediate markup language to do the job. So I setup and tried two lightweight markup language for writing blog posts.

reStructuredText

reStructuredText has been officially accepted as the Python inline code documentation formatter. It currently doesn’t handle code formatting, but it’s in the works. This fellow notes that Pygments already has a way to integrate with reStructuredText. It took a lot of finagling (and realizing my PATH was wrong), but I got it working.

Here’s what I did (perhaps not the best way)

Download Docutils and unpack it.
Run setup.py build which creates a build directory
Move all the files in the build/scripts-2.5 directory to the build/lib directory
Running rst2html.py will now work, but it can’t do highlighting yet
Download the Pygments source and .egg file for the proper Python version
Randomly run a bunch of commands on the Pygments source and/or .egg until it somehow gets installed as a Python lib (yes, that’s pretty much what happened)
Find the rst-directive.py file amidst the Pygments source, copy its contents, and paste it into rst2html.py right before the last line of that file.

reStructuredText will now recognize the sourcecode directive. There are other options you can set in that Pygments directive code (such as enabling the :linenos: option).

asciidoc

asciidoc works right out of the box, but needs GNU source-highlight for code highlighting (although I guess you could hack it to use whatever else you want). You can download a very old Windows binary, but the latest version has many more highlighters that aren’t compatible with older versions.

Since source-highlight needs some of the boost libraries. I didn’t have much hope in building it with MinGW, so I just went with the Cygwin build. The build process is pretty painless:

Using the Cygwin setup, download the boost libraries under the Devel category.
Downloaded the latest release of source-highlight
Unpack it into a path without spaces (the build with fail otherwise)
Fire up the Cygwin shell and cd into the directory
10+ minutes of ./configure, make, and make install, and you’re done

Running asciidoc.py with the -s option removes the HTML header/footers, and source-highlight inlines all its CSS with the code. After getting it working, I recreated the last post in asciidoc format, and replaced the post with the generated output.

It looks really nice.

Blogging with asciidoc

All-in-all, asciidoc is very usable and powerful, but it does a lot more than I wanted, and the syntax isn’t as natural as I would like.

By default asciidoc and the blogspot styling don’t completely get along. The listingblocks don’t have the overflow: auto style, and it takes some tweaking to fix it for both Firefox and IE. There is also a lot of extra spacing added due to several layers of tags, but that’s not hard to fix.

I started an asciidoc syntax configuration for jEdit.

I wonder if there’s a way to create a tool like this for blogspot.

Comparison of Hibernate with Netbeans and Eclipse

2009-04-20T13:20:00.000-07:00

Netbeans 6.5.1 with Hibernate Support plugin 1.0.3

Update: Looks like the new Netbeans 6.7 has really improved Hibernate reverse-engineering support.

Netbeans’s Hibernate implementation is pleasantly simple, however its reverse-engineering is currently incomplete. If you want a straight-forward introduction to Java and Hibernate, Netbeans is the perfect way to go. But as soon as you want more powerful visual reverse-engineering tools, you’d best switch to Eclipse.

Steps to use Hibernate in Netbeans

Add Hibernate Support plugin
Setup a database connection in the Services window
Create a new Java Application project
Create a new Hibernate Configuration File (hibernate.cfg.xml)
- Select the database connection
- press Finish

Edit the hibernate.cfg.xml and add these properties

hibernate.show_sql = true # under Configuration Properties (optional)
hibernate.current_session_context_class = thread # under Miscellaneous Properties

Create new “Hibernate Mapping Files and POJOs from Database”
- select the configuration file
- choose the tables to reverse engineer
- check “JDK 5 Language Features” to get type-safe generics
- select a package where reverse engineered classes will go
- press Finish

Create new “HibernateUtil.java”

So Hibernate can find the configuration file even if it’s in the jar, I recommend changing the line

sessionFactory = new AnnotationConfiguration().configure().buildSessionFactory();

URL cfgurl = HibernateUtil.class.getResource("hibernate.cfg.xml");
sessionFactory = new AnnotationConfiguration().configure(cfgurl).buildSessionFactory();

There’s currently no way to regenerate the reverse-engineered code once it’s been generated. This thread says it is “targeted as part of the next release” (Netbeans 6.7 in June '09).

Eclipse 3.4.2 with Hibernate Tools addon 3.2.4

Is it just me, or is it really hard to find up-to-date information on the web about setting up Hibernate in Eclipse? There is one tutorial that doesn’t seem too old, and covers most of my steps listed below.

Being new to Eclipse, it might be difficult to figure out there are actually 2 different addons you can download: Hibernate Tools, and Hibernate Synchronizer (last release 3 years ago?). There’s also MyEclipse that costs money but probably makes the whole process more seamless. To get started in Eclipse, it might be useful to understand how Eclipse does Java.

There are 4 different configurations to be aware of for Eclipse Hibernate Tools:

The Configuration file (hibernate.cfg.xml), used by the Console Configuration (technically optional because most if not all of these settings can be made in the Console Configuration)
The Console Configuration, used when creating the Reverse Engineering file, and by the Hibernate Code Generation configuration
The Reverse Engineering file (hibernate.regeng.xml) used by the Hibernate Code Generation configuration
The Hibernate Code Generation configuration

So with plain Eclipse, here’s how you setup Hibernate Tools.

Find the URL to the Eclipse Hibernate Tool addon
Add the site to the Software Updates and install the addon
Find all the libraries necessary to use Hibernate in your program and download them
Create a new Java Project
- under the Libraries tab, add all those Hibernate libraries
Open up the project properties
- in Hibernate Settings, check “Enable Hibernate 3 support”
Create a new Hibernate Configuration file (hibernate.cfg.xml)
Note that this configuration file is only to be used by the Console Configuration
- select database dialect
- select driver class
- enter the connection URL
- check “Create a console configuration”
- press Finish

Edit the hibernate.cfg.xml and add these properties

<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.show_sql">true</property>  <!-- optional -->

Create a new Hibernate Reverse Engineering file
- select the Console Configuration you just made and press Refresh
- select which tables to reverse engineer
Create a new Hibernate Code Generation configuration
- select the Console Configuration
- select the Output Directory (if the output directory contains your hibernate.cfg.xml, it will be overwritten by this task)
- check “Reverse Engineer from JDBC Connection”
- enter the regeng.xml file you created
- under Exporters tab
  - check “Use Java 5 syntax” to get type-safe generics
  - check the following Exporters: Domain code, Hibernate XML Mappings, Hibernate XML Configuration
- press Run
At this point your original Hibernate Configuration file could be replaced by the new one created by the Code Generation, or you can keep them separate

There’s no HibernateUtil generator, but all you really need is this

URL cfgurl = this.getClass().getResource("hibernate.cfg.xml");
SessionFactory sf = new AnnotationConfiguration().configure(cfgurl).buildSessionFactory();

In my specific case, I’m using Oracle 10g and the tables are in a schema named anotherschema. There are some things to watch out for.

Add the Oracle jdbc driver jar to your project libraries.
Case sensitivity!

Netbeans:

Add the schema to the database connection properties after it’s created (this also means each schema needs a separate connection).
Has a problem that prevents me from reverse-engineering a database more than once without restarting the IDE (error says table “cannot be added because it does not have a primary key”).

Eclipse:

Most places that need to read or refresh the database schema will often freeze Eclipse.

Installing Linux on a Toshiba Satellite T1960CS

2009-04-10T20:41:00.001-07:00

Some time back in 2007 I had the notion to install Linux on an ancient laptop I had sitting around. I figured it might be useful as a resident machine for torrents, a small web server, or even a router.

This was originally posted on a page at atwiki.com, but somehow got deleted. Thankfully archive.org had picked it up, so here it is again, all cleaned up.

The Computer

Toshiba Satellite T1960CS

486, 66Mhz
20MB RAM
600MB HD
VGA display
PS/2 Keyboard and Mouse ports
3.5'' floppy drive
Serial Port
Parallel Port
PCMCIA Ethernet IC-Card+

No CD drive. No USB port.

This computer also has a known habit of not liking certain types of boot floppies. The floppy will be perfectly fine, and will boot on any other computer. Distasteful floppies make the Toshiba respond with the very detailed message of: “Boot failed”. The Goal

Install some Linux onto the Toshiba T1960CS

The approach suggested in the Damn Small Linux page:

Find a tiny distro that can boot from floppy disk
Ensure the tiny distro can enable interweb abilities
Download and install the real distro of choice to the lappy

Tiny Distros

DSL suggests TOMSRTBT. I never figured out how to enable PCMCIA with that distro (maybe some guru knows how). DSL also suggests BGRescue, but that floppy gave a “Boot failed”.

Other tiny distros tested:

http://www.xinilux.us/olaf/ ← “Boot failed”
http://www.xinilux.us/portable_coyote/ ← “Boot failed”

Finally: http://distro.ibiblio.org/pub/linux/distributions/baslinux/

Boots! detects network card (with some tweaking)!

BasicLinux

Boots from 2 floppies. First floppy can be made with simple dd command. Second floppy needs to be formatted MS-DOS, and you copy the tar.gz file onto it.

This thing actually has an X server, with some very basic applications.

After booting with the first disk, do the following:

1) Partition and format the hard drive

I chose a main partition, and a 64mb swap partition.

$ fdisk /dev/hda
: d (delete existing partitions)

: n (new partition)
: p (primary partition)
: 1 (partition 1)
: <Return> (first cylinder)
: +636M (last cylinder)

: n (new partition)
: p (primary partition)
: 2 (partition 2)
: <Return> (start where partition 1 left off)
: <Return> (to the end of the disk)

: a (make a partition bootable)
: 1 (partition 1)

: t (type of partition)
: 2 (partition 2)
: 82 (hex code for swap partition)

: p (check that the partitions are correct)

: w (write the partitions)

2) Format the partitions

$ mke2fs -c /dev/hda1
$ mkswap -c /dev/hda2

3) Mount the drive

$ mount /dev/hda1 /hd

4) BasicLinux to HD

install-to-hd

reboot the system with floppy 1 in the drive
press CTRL when you see LILO
enter this: hd root=/dev/hda1
execute: lilo -v
get the bas2hd package
insert the disk, copy bas-hd.tgz to /tmp, and run go
Edit /etc/rc.hd

Now that I have a drive ready and waiting, I need to get to the interweb.

$ edit /etc/rc.hd

Comment out these two lines

e2fsck -pf /dev/xxxx 2>/dev/null
mount -avt nonfs

Uncomment this line

# /etc/pcmcia/start

Add the following lines at the end (self assigned IP and netmask, default gateway, your DNS servers)

ifconfig eth0 192.168.###.### netmask 255.255.255.0
route add default gw 192.168.###.###
echo "nameserver ###.###.###.###" > /etc/resolv.conf
echo "nameserver ###.###.###.###" >> /etc/resolv.conf

Exit and save with Ctrl+X and y

Copy this new file to the hard drive or floppy to save it

$ cp /etc/rc.hd /hd

Then run the script

$ sh /etc/rc.hd

And make sure the internet is actually working

$ ping google.com

I had to change my pants when it worked the first time.

Get the real distro: DSL…or not

My first attempt at a distro was DSL. I followed their instructions for the “Poorman’s install”.

Unfortunately BasicLinux doesn’t have ash installed, so it can’t run the frugal_lite.sh script. But looking at the script code, all it does is this:

download the current DSL iso
mount it
copy everything out of it, onto the harddisk
download the floppy boot image
dd the floppy image to a floppy
reboot the computer

More unfortunately, I never could get BasicLinux to mount the iso.

Sooo…on another computer, I had to take all the files out of iso and put them into a tar.gz, then run a quick web server with newlisp (newlisp -c -d 8080 -w), so the lappy could wget the file (there’s gotta be a better way to do this).

Finally I got all of the DSL iso files copied onto the lappy harddrive.

I made a boot disk with dd from the ``bootfloppy.img'' file. Unfortunately the Toshiba didn't like the boot disk (“Boot failed”). After confirming that it just wasn’t going to take that image, I instead tried bootfloppy-grub.img. I had to change my pants again when it actually did boot.

Unfortunately, neither of the two Grub options worked. So I had to manually tell it. Press 'c' for the prompt, and type these commands (from here)

root (hd0,0)
kernel /boot/isolinux/linux24 root=/dev/hda1
initrd /boot/isolinux/minirt24.gz
makeactive
boot

(the command "title DSL" just gave me an error)

One last pair of pants for when this thing actually started booting up…and I had plenty of time to change because the bootup took forever.

DSL saw I had PCMCIA, and even the network card, unfortunately it said it didn’t have the drivers for any of it. Also my mouse wasn’t working.

At this point I give up on DSL because I didn’t want to figure out how to install the drivers for my network card without having a network card to download them (I’d have to find them on another computer and copy them onto a floppy?).

Another distro?

I thought I might also try Feather Linux. I have plenty of disk space, and this slightly bigger distro might have my network card drivers.

Boot without CD
http://featherlinux.berlios.de/booting.html

Need to download the USB distro zip, and unpack to the drive. Make the boot floppy. Once booted, I should be able to install it.

Other link
http://featherlinux.berlios.de/phpBB2/viewtopic.php?t=994

Not to be continued

Got burned out with all this madness. If I ever resume trying to get Linux installed, I plan to just use BasicLinux-at least I know it works. Plus I found X detects the mouse so long as it’s plugged in during bootup and never removed.

Unfortunately BasicLinux doesn’t come with gcc, so adding that would be the first priority.

Miscellaneous

Linux PCMCIA Info
http://pcmcia-cs.sourceforge.net/ftp/doc/PCMCIA-HOWTO-3.html
http://www.linux.org/docs/ldp/howto/Ethernet-HOWTO-4.html

Windows driver for the PCMCIA Ethernet card
http://members.driverguide.com/driver/detail.php?driverid=31708

A fine transform

2009-04-10T13:53:00.001-07:00

Spent a day trying to figure out why this function worked…

void transformDraw(BufferedImage bi,
                   Stroke st, Color c,
                   AffineTransform at, Shape sh)
{
    Graphics2D g2d = bi.createGraphics();
    g2d.setStroke(st);
    g2d.seteColor(c);
    g2d.draw(at.createTransformedShape(sh));
}

…but this one…

void transformDraw(BufferedImage bi,
                   Stroke st, Color c,
                   AffineTransform at, Shape sh)
{
    Graphics2D g2d = bi.createGraphics();
    g2d.setStroke(st);
    g2d.seteColor(c);
    g2d.transform(at);
    g2d.draw(sh);
}

…always resulted in the BufferedImage being filled entirely with the stroke color when I called the function like this…

transformDraw(myBufferedImg,
              new BasicStroke(3), Color.RED,
              AffineTransform.getScaleInstance(100000, 100000),
              myShape);

It was this pdf that helped me finally understand why those two functions are different.

Clever RPG Puzzle?

2009-03-02T23:26:00.000-08:00

Or terrible pun?

World of Warcraft's Poor Security

2009-02-27T20:18:00.000-08:00

By default, World of Warcraft fails to run properly under a limited account. The game will start and run fine, but no settings are saved (e.g. UI, setup, or addon changes) because everything is stored under the folder that WoW is installed (Program Files) which should be read-only. A reasonable work-around might be to make some of the subfolders under the WoW folder writable (such as WTF, Screenshots, etc.).

A new failure of WoW security design was recently added with the game launcher. On every single run, the game launcher seems to randomly generate a brand-new exe at some random place on your computer. Firewalls are about as thrilled by this as I am. It defeats any kind of whitelist security, and a game has no business complicating computer security like that.

Additionally, there is an uncommon trick World of Warcraft uses that blocks other programs from inspecting its process in memory. This can also prevent other programs from forcefully shutting it down. The technique has to do with Window's Discretionary Access Control List (DACL). It is related to the security tab you see on Windows XP Professional that defines who has what access to various parts of your computer.

A processes's security limitations can be changed dynamically during runtime by the process itself, and that's exactly what WoW does. A process changing it's own security at runtime is not particularly valuable because any other process with equal or greater security access can change it back. It's also meaningless when running as an administrator because every process automatically has all access to everything.

None of this is new. People have known that WoW does for years now. I even ran across a small bit of code specifically to adjust the security of WoW. I've updated it bit to simply remove WoW's security settings then exit.

I'm new to this whole DACL business, so I might have missed some things.

Warning: Any use of this information, whether benign or malicious, might result in having your World of Warcraft account terminated. Take heed, lest you anger the Blizzard gods.

Locking down my Windows XP machine

2009-02-22T23:22:00.000-08:00

During my last Windows XP re-install, I pondered for a moment if I should use proper computer security by running within a limited user account. Eventually my desire for convenience, and realization that I wield the raw power of the OS far too much, kept me with a default admin account.

I’ve never really needed much protection. I can count on one hand the number of viruses my antivirus has found on my computer this decade. Only one of those tricked me to execute it. Over time, my weekly antivirus false-positives caused me to disable the nightly scans.

No, my fear of unwanted programs wasn’t enough to drive me into putting a lock on that admin door. It turned out to be the fear of programs I do want that pushed me to the switch. It started with World Of Warcraft’s Warden, and culminated with Mirror’s Edge’s SecuROM. Invasive DRM and rootkits concern me far more than any virus.

But I’m glad I didn’t try to live in the limited account back then. It would not have lasted very long before I would be sick of always having to switch to an admin account whenever I wanted to burn a CD, change anything in the control panel, install anything, or modify system or program files. There is the "Run as…" option, but that often leaves changes made to the account being "run as", instead of the current account.

Luckily my Linux experiences in the last year taught me that you can have the best of both worlds with a program like sudo. My searches for something like that for Windows led me to 3 open source solutions.

SudoWin (Sudo for Windows): This is repeatedly recomenneded by readers of Atwood’s Real Ultimate Programming Power blog.
suDown (or sudoWn): Tried briefly before running across…
SuRun: Seems to be the new kid on the block. I have been using this one for several weeks now, and so far I love it.

For better or for worse, I disabled having to actually enter a password. This was mainly due to my need for some programs to always run with admin access (especially at startup). For everything else, I get a nice popup that just needs an OK click, not much different from Vista’s UAC. Update 1: The new v1.2.0.6 lets you skip password input for your white-listed apps if you want.

Update 2: After using SuRun even longer now, I’ve only found 2 problems with it: (1) When browsing explorer as administrator, any changes you make don’t get reflected in the file browser (you have to manually refresh to see), and (2) It sometimes takes several seconds to open new Windows explorer windows. In general big delays have occurred for any kind of file system browsing, or low-level associated functions. For example, mounting/unmounting a TrueCrypt volume can take up to 10 seconds, and has even failed with a strange error on unmounting at times.

Update 3: Found an insightful thread where someone discussed with the SuRun creator about some aspects of the program.

It’s kinda comforting to see Process Explorer unable to view any details of admin processes. I know that Warden and SecuROM* will be hitting the same roadblock.

However I ran across something a little disturbing with World of Warcraft the other day. Process Explorer was unable to view details of the Wow.exe process, as if it was running as admin without my knowledge.

To be continued… * I’m aware that SecuROM has been known to install a service, thus always granting it admin access, but I’ve kept a watchful eye on my services and haven’t noticed anything yet.

Faith Model Continued

2009-02-15T22:26:00.001-08:00

I spent several more hours examining the CH_Faith_Cinematic.upk posedAgain StaticMesh. First off, I discovered that my raw data exporter in UE3PackageViewer doesn't seem to work correctly (it cut off half of some of the data). Manually pulling out the data, I found several lists of structures:

struct UnknownStruct1 {
    struct Vertex1 {
        float x;
        float y;
        float z;
    };
    struct Vertex2 {
        float x;
        float y;
        float z;
    };
    int UnknownInt;  /* either 0 or 1 */
    union {
        struct { /* when UnknownInt == 0 */
            short int UnknownStruct1Index;
            short int UnknownStruct1IndexPlus1;
        };
        struct { /* when UnknownInt == 1 */
            short int Count;
            short int TriangleIndex;
        };
    };
};

struct Triangle {
    short int VertexIndexes[3];
    short int UnknownShort; /* material number? */
};

struct Vertex {
    float x;
    float y;
    float z;
};

struct UnknownStruct2 {
    char Unknown1[4];
    char Unknown2[4];
    int NegOne; /* always -1 */
    char Unknown3[4];
};

struct UnknownStruct3 {
    int UnknownInts[4];
};

There is also a large list of shorts and another of bools.

The list of Vertex and Triangle work out to the model in my last post. Of the rest, the list of UnknownStruct1 is the most interesting. Those verts aren't normals or texture coordinates (nearly all are > 1.0). When visualized in space, they hover all around the Faith model.

Additional searching about Unreal meshes led me back to the UnrealToX3D converter. The author wrote a nice white paper about converting the t3d format.

The real challenge was converting the texture coordinates. Unreal textures are described as tiling a plane in space coincident with the polygon on which they lie.

Each surface has a 3D vertex for U, another for V, and another for the texture normal (these are usually orthogonal). A pan value (in UV units) may also be supplied. This scheme allows for arbitrary scale, rotation, shear, and displacement of the texture without having to describe UV coordinates for each vertex on a polygon. To turn these various directionals into common UV coordinates, I find a transform that rotates the polygon from its arbitrary orientation onto the XY plane, and apply the same transform to the U and V texture components. I can now use the transformed U sand V component (which are now co-planar with the XY plane) to scale the texture. Using the pan components as offsets, I now use the polygon's coordinates (which are now all in the XY plane) to produce the UV coordinates.

Could those random points in space actually be 3D UV coordinates? I looked at the faces associated with the points, but the relation doesn't make any sense to me.

I wrote an exporter for this model and structures in Python if you want to look at the data yourself.

Other searching led me to comments in forums that say you can convert StaticMesh to brush in UnrealEd, and then export it as OBJ. I tried poking around but haven't quite found how.

Unreal 3 Engine Model Extraction

2009-02-07T22:21:00.000-08:00

While there is a lot of modding that may be possible with Mirror's Edge, my primary desire is to gain access to the various models of the game.

I have searched high and low, with at least 30 different queries thrown at Google, and several hours of scanning results, but nothing has turned up. No one seems to have a way to extract models from the Unreal 3 Engine. Which suggests that this is the first time anyone has.

If someone has already found a way to extract models, then I'd be more than happy to be wrong. Thus far my mesh explorations have been hit and miss. This Faith StaticMesh resource (CH_Faith_Cinematic .posedAgain) has easily identifiable vertices and triangle indices, but I can't locate the texture coordinates. The resource right next to this mesh (posedAgain_Shaded) is even less clear. The data is tremendously random, with no obvious patterns. The S_M95 weapon StaticMesh appears to be the same.

After several hours of dead-ends I grew frustrated, so I played Unreal Tournament 3 for the rest of the day :) I'll give it another go eventually, but I don't have much hope in the results.

Update: Gildor's Unreal Mesh Viewer has now been updated to extract Mirror's Edge SekeletalMeshes. These are different from StaticMeshes (like the one above), and they include the bone structure of the model as well. The Faith SkeletalMesh (the one you actually see running around in the game) is in CH_TKY_Crim_Fixer.upk as SK_TKY_Crim_Fixer.

In Capitalist America, DRM messes with YOU

2009-02-03T19:46:00.000-08:00

Suddenly had an itching to play Gears of War today for no reason. Stupid #%^$@$## DRM.

Edit: Since the source of the problem is due to the unintentional expiration of the programs digital signature, which was used solely to prevent people from playing PVP with modified executables, there has been a ton of debate whether this is really DRM.

I assume most people consider DRM the new lingo for 'copy protection' (which I think was a more accurate description). This interpretation of DRM obviously does not apply to the core problem above. But given what Digital Rights Management describes (it manages your right to use their data), then I'd say this is certainly DRM. I paid for the game, but I don't get to do whatever I want with it.

On the plus side, maybe the patch they'll probably release will include at least a few other bug-fixes for this crap-tastically buggy game.

Messin' with SecuROM

2009-02-01T14:43:00.001-08:00

Process Monitor is one of the programs that SecuROM blacklists. If you try to start Mirror's Edge with Process Monitor running, you get a beautifully designed^*, and clearly explained^* dialog that looks like this. * /sarcasm

But it's not hard to run Process Monitor with Mirror's Edge. Login as an administrator and startup Process Monitor. Then switch users to a limited account and run the game. Having done this with Mirror's Edge, I can see it creates a dll in your Temp folder called drm_dyndata_#######.dll. The '#######' seems to always be the same number for me, but other people report a different number on their computers. If there is some kind of 'error' (i.e. they want to have more control over your computer than you do) and Mirror's Edge refuses to start, then it also creates a dll called drm_dialog.dll.

The Turtle's Blog Presents: Messin' with SecuROM

Just for fun, I set the security of my temp folder to WRITE-ONLY and started up Mirror's Edge. Well SecuROM sure didn't like that, and somehow concluded that my game is no longer registered, and began its phone home process.

Next I went to the SecuROM registry folder and exported the entries, then deleted them from the registry. Starting up Mirror's Edge, I blocked its attempts to phone home for a new registration. I then restored those exported SecuROM registry entries, and thankfully SecuROM was appeased and my game ran fine again. Note that Mirror's Edge has to be separately activated for every account on your computer. I tried importing these registry entries on another account, but SecuROM rejected it and tried phoning home.

Mirror's Edge phone home process involves trying to connect first to EA (proxy.novafusion.ea.com), then Google (www.google.com) with three different IP addresses, and finally Microsoft (www.microsoft.com) with two different IP addresses. It gives no indication it is doing this, and if Mirror's Edge fails to make contact with any of its desired sites, it takes about two minutes before finally reporting it needs internet connection. I'm sure there are some oblivious users out there that double-clicked that icon like crazy, and after seeing no response, concluded that there is something wrong with the game or their computer.

I thought I would try being one of those oblivious users, and started Mirror's Edge on an unregistered account--then kept double-clicking the icon until my computer froze briefly and started beeping at me. I had some 15 MirrorsEdge.exe processes started and my CPU maxed. In a few seconds all but the original process closed, but still no indicator that MirrorsEdge.exe was really doing anything until the eventual message is displayed.

When will these game studios learn that this is (one of the reasons) why people hate DRM--because it's implemented so very poorly. I submit that a big reason Valve's Steam has finally been somewhat accepted is because they try to reduce the hassle as much as possible. But I'm sure the rantings on some no-name blog amounts to a hill of Circuit City stock in the eyes of Sony.

Disclaimers: Don't try these sorts of things unless you know what you are doing, are willing to possibly lose one of your 5 registrations, and don't mind if your computer could crash beyond repair. I have a fully legal copy of Mirror's Edge, and was perfectly happy to support the developers that did such a great job on the game. I wish more people would do the same.

Stupid Dragon

2009-01-29T20:01:00.000-08:00

Ever since patch 3.0, a terribly annoying dragon resides in the World of Warcraft login screen. Every couple of minutes it lets loose an obnoxious roar.

You are sure to always find at least one active thread in the WoW forums focusing on this terrible design flaw. But it astounds me that the replies are mostly to tell the poster to STFU, or suggest half-baked solutions to what is the real problem: a stupid dragon. For those of you that find the 'whiners' more annoying than the dragon, I recommend these macros you can play every time you run across one of those threads:

/script PlaySoundFile("Sound\\Ambience\\GlueScreen\\Wrath_Login_1.wav")
/script PlaySoundFile("Sound\\Ambience\\GlueScreen\\Wrath_Login_2.wav") 
/script PlaySoundFile("Sound\\Ambience\\GlueScreen\\Wrath_Login_3.wav") 
/script PlaySoundFile("Sound\\Ambience\\GlueScreen\\Wrath_Login_4.wav")

For those of you who would rather spend time getting rid of the roar, the fix turns out to be terrifically easy.

Download this file and put it into your World Of Warcraft/Data folder to shut the dragon up.

How this works (I think): When WoW needs a resource (a sound, graphic, etc.) it checks the name indexes found in the various MPQ files. If two MPQ files contain the same named resource, it picks one of them (probably from the last file in alphabetical order). So this MPQ file essentially replaces those 4 sound resources above with empty sounds.

Don't believe me? You can use Ladik's MPQ Editor to take a look at the MPQ file.

I cannot take credit for this hack. Someone in the WoW forums posted this fix in a thread I am now unable to find again.

Dissecting Mirror's Edge

2009-01-28T22:14:00.000-08:00

I'm brand new to the inner workings of the Unreal Engine, but this is what I've found works for Mirror's Edge.

Enabling the built-in Mirror's Edge editor!: Folks have gotten the real Unreal Editor mostly working for Mirror's Edge.

UE3PackageViewer: Hands down the nicest and completely open source package viewer found. I've already updated it to work with Mirror's Edge. It can preview DDS images (double-click the Texture2D in the detailed list), and even read compressed packages (.me1 files). I've also added Ogg extraction, and raw dump extraction for unknown data (somewhat broken). WARNING: the program is still full of bugs (and I don't have much motivation to update it any more that I have).

Upk Info: Not quite as extensive package viewer, and doesn't currently work with Mirror's Edge, but it's also open source.

Unreal Model Viewer: Soon to be open-sourced model viewer that works for many Unreal Engine games, including UT3 and Mirror's Edge (Gildor is the man).

Oggextract: Another open-source tool that scans and extracts Ogg files from the packages.

Another Ogg extractor: Closed source Ogg previewer and extractor.

NVidia's content creation tools: Includes DDS image viewers and converters.

Bink RAD tools: To watch and convert all the videos of the game.

3D Ripper DX: Can capture models being rendered by DirectX 9 on Windows XP.

Documentation

The Beyond Unreal Wiki: Documentation for practically every aspect of older (<= 2004) Unreal Engines.

Mod Authoring for Unreal Tournament 3: Serious documentation is serious.

Other points of interest

UShock: Open source UT2004 level viewer. Doesn't work with Mirror's Edge, but maybe it could be updated to do so.