Wednesday, April 22, 2009

Join Progress

I've worked on the Joining with attributes a little bit but I've run into a problem. Maybe it's not really a problem but I just think it is.

I can easily get the Join string from the attribute, and the new .WhereJoin uses it. The method gets all the tags for the user as expected.

So whats the problem?

The problem is that I have to specify the column instead of a property. I don't like that because it means I have to know the column name. On the other hand, I know all the column names because I set them myself, so it's not terrible

It still bothers me though.

Wednesday, April 15, 2009

Join is working

I have the join working but it's not pretty.

_tags = DBObject.WhereJoin("user_id", "=", this.ID, typeof(DBTag), "tag_user", "tag_id", "id", DBUser._conn);


Right now, in the Where you have to specify all the information about the Join, the table, the columns, etc. and it doesn't use a property but the column name (because there is no object for the join table.)

I think I'm going to try and move some of this information into the attribute. I also noticed that I need something like a Helper object, or something to help me get information about the object and prevent from calling the same methods over and over again every time I need the same bit of information.

For the Join, I need to know:
  • Join Table name
  • Join column on the join table.
  • Join column on the normal table
  • Where Column on the join table
  • Where value

Do I want a static Search column for the join? Or do I want to be able specify that in the method?
In my example:
SELECT * FROM tag JOIN tag_user ON tag_user.tag_id = tag.id WHERE tag_user.user_id = 2;

This gets all the tags for the user, so this will always be the case. The easiest would be to assume this is the default until I have evidence to the contrary.

Monday, April 6, 2009

Back to bussness

Ok, so My top pirority for this right now is to get a list of object via a join table.

I have the Tag Object, which has 'id' and 'name'.
There is a join table tag_user to Join Users to Tags.

I need to pull a list of tag via tag_user.

Friday, April 3, 2009

LINQ

I've been avoiding LINQ for several reasons. One is that I'm afraid that it would mean all my code and effort went wasted. (Not a good reason I know.) Another was that it doesn't really work with MySql yet. I know there are ways to make it work, but it's not production ready or easy to use.

I read an article about LINQ which was very informative. The bottom line seemd to be that LINQ was great for small databases that didn't change offten. But once you started having large databases or complex data structures it becomes a hassle.

Another article seems to share some of my conserns with LINQ.

But these articles are old. Maybe I'm just looking for something to be wrong with LINQ. My frist reason for not using it so far can be a pretty powerfull modivator. Maybe I need to just give it a go.

idea

An idea has struck me. What if the DBObject could build its self from the database? You specify the table and it would take care of all the properties and accessors. The one problem with this would be that you lose type safety and intellisence features. The programmer would have to know what columns exist in each table without intellisence there to remind them. There would still be the join problems, but I'm already trying to work those out.

This kindof design would be more data driven and easer for other developers to use.

How do I tell the the object what type each column is? I don't. The developer would have to know this on their own. In a static typed language like c#, this could be very frighting.

Friday, March 20, 2009

On the right path.

So I made an Index version of the DBObjects to test this new method. To say the least, it's been a very inlighting. 

For my test each object had to get 1,248 User records 100 times (so I could get an average). As far as just retraving the inital list, the index method worked considerablly faster. This didn't come as a surprise because it's just geting all the IDs for all of the user records in the system.

Normal method Retreave List: 3.8 seconds
Index method Retreave List:    0.43 seconds

As I said this isn't surprising because the index method only fetches 1 out of 5 columns. Now on to actually accessing a property for every item in the list

Normal method Access: 0.0 secionds (around 6,700 ticks)
Index method Access:    3:50.50 minutes (around 850,000,000 ticks)!

The index method lost horribly! It just takes too long to query the database 1,248 times. So far this is pretty much what I expected. Heres the total averages.

Normal Retreave & Access: 2.13 secionds (21,380,733 ticks)
Index Retreave & Access: 1:53.21 minuetes (around 1,132,100,354 ticks)

So far it looks like I've been on the right track. In my experiment I did create something neat. For the index I created the abliblity to get a list of objects from the DBList object, with a call like DBList users = new DBList(typeof(DBUser));

One thing I might want to try, is instead of making 1,248 calls to the database, to get batches at a time. So if I want user.id = 1, it'll actually load the first 100 or so items. That would reduce our class to 13. But still I just don't see how 13 calls is going to be any faster than our single call to get all 1,248 items.

Speed Questions

I've been messing around with Objective-C and developing in OSX. So I was reading a tutoral using SQLite3 and the tutoral did something I'd never seen or thought of with the database object. He had the list object populate only the ids (index/primary key) of the objects. Then when you tried to access any of the properties on the object, it would use that id to fetch the rest of the object.

I was intreged, this is better? I had assumed that I would want to get as much data as possible in the fewest calls as possible. But I never actaully tested this theory (or have any idea where it came from.)


So I'm going to test this. I'll perform three tests.

  1. I'll try the way I've been developing the objects. Where I try to get as much informaiton in each SQL statement as possible.

  2. I'll use the method I read about in the tutoral, where I'll get the ids from everything in the list and return a completed object as I call it.

  3. I'll try a varation of the new method. I'll get all the ids, but I'll only return the property requested as it's requested instead of filling out the whole object.


The database is MySql 5.1 with InnoDB tables

Wednesday, March 11, 2009

Difficulties

I've been having problems completing this next step. I think I've been trying to solve two difficult problems at once. The Dynamic loading of the property, and the joining of the two tables.

So I'm taking a step back. Let's solve the join first. How can I solve this problem with the code I already have?
Well I could just manually make the call to Tag in the User property. I would need to use the .Where() method to get what I want.
But, the .Where() method only looks at the objects table. I need to reference a joining table (tag_user to be exact.)

So, I need to be able to specify a joining table in the .Where() method, and then pull normally. I need to identify the table name, the matching columns, and then do the .Where() method normally.

Sounds good, I'll get on this in the morning.

Friday, March 6, 2009

How

I changed some things around. I added a GetProperty and SetPropery because I didn't like have to do a check all the time. I think it simplifies things and allows _properties to be private instead of protected.

I also created a GetColumns that gives the full name with an optional prefix for select statements. I created a GetTable to help out with this as well.

Wednesday, March 4, 2009

helpers

I which I could put this in the base class somehow, but we can simplify the code if the inherited class override the By and Where methods like so


///
/// Get a single User By the property and value specified
///

///
///
///
static public DBUser By(string property, object value)
{
return (DBUser)DBObject.By(property, value, typeof(DBUser), DBUser._conn);
}

///
/// Returns DBList where all the DBUsers have a matching property
///

///
///
///
///
static public DBList Where(string property, string evaluator, object value)
{
return DBObject.Where(property, evaluator, value, typeof(DBUser), DBUser._conn);
}


Then we can make simple calls like:

DBUser u3 = DBUser.By("firstname", "chris");
//and
DBList users1 = DBUser.Where("id", "<=", 1313);

Next Step

Well I guess the next thing to do is to allow an object to relate to another object. For work I use tags for objects (as I discussed earlier) so that's going to be my starting point.

So I have two objects. User and Tag.

A single User record can have an unlimited number of tags.
A single tag record can have only one user. But a Tag object can have an unlimited number of users.

So we have a many to many relationship, with a table joining them. The table already exists (tag_user) so I need to specify that.

When I get a user, it should get all of his tags as well, like wise when getting a tag it should get all the users. except that might not always be a good idea. What if I have 10,000 users with the tag 'Public' I might want to get that tag without getting all the users who have that tag. alsoIf i'm getting a list of users and they all have the 'Public' tag, it shouldn't the tag shouldn't pull that list of 10,000 for each user in my list. It should only pull it once, and thats only if I ever request it.

So I need to do some lazy loading. Say it only gets the related object if it's requested. So if I get a list of users, it'll just fill out that list. If I access the tags on one of the users, then it'll go and fetch his tags.

So first things first:


Create Many to Many relationship, if you access the related object, then it goes and gets it.

Monday, March 2, 2009

DBObject.Where

Ok it works. I also created a checker for the opertators so we don't get invalid/malicious code.


///
/// Returns a DBList of objects that match the Where
///

/// Property to Match
/// "=", "<", ">", "LIKE"
/// Value to Match
/// Type of the object to get (Must inherent from DBOBject)
/// Connection String to use
///
static protected DBList Where(string property, string evaluator, object value, Type who_type, string connection)
{
DBList list = new DBList();

//Find the Column Name and check that we have everything, if fail return the blank list
string column_name = DBObject.GetColumn(property, who_type);
if (column_name == string.Empty || evaluator == string.Empty || value == null) { return list; }
//Check the evaluator, If fail, Return a blank list
evaluator = DBObject.CheckEvaluator(evaluator);
if (evaluator == string.Empty) { return list; }

//Create the WHERE
string query = DBObject.SelectQuery(who_type) + " WHERE " + column_name + " " + evaluator + "?value";

using (MySqlConnection conn = new MySqlConnection(connection))
{
MySqlCommand cmd = new MySqlCommand(query, conn);
cmd.Parameters.AddWithValue("?value", value);

try
{
conn.Open();
MySqlDataReader reader = cmd.ExecuteReader();

while (reader.Read())
{
DBObject obj = (DBObject)DBObject.FromReader(reader, who_type);
list.Add( obj );
}

conn.Close();
}
catch (Exception ex)
{
Console.WriteLine("\n\nERROR:\n" + ex.Message);
}
}

//No matter what happens, always return a valid list
return list;
}


///
/// Accepted Evaluators: '<', '<=', '>', '>=', '=', '!=', 'LIKE', 'REGEXP', 'IS NOT'
///

///
///
static protected string CheckEvaluator(string evaluator)
{
//Lower and trim
evaluator = evaluator.ToLower().Trim();

switch (evaluator)
{
case "<":
return evaluator;
break;
case "<=":
return evaluator;
break;
case ">":
return evaluator;
break;
case ">=":
return evaluator;
break;
case "=":
return evaluator;
break;
case "!=":
return evaluator;
break;
case "like":
return evaluator;
break;
case "regexp":
return evaluator;
break;
case "is not":
return evaluator;
break;
default:
return string.Empty;
}
}


And an example:


//Now lets try to get a list of users
Console.WriteLine("\n\nTrying the Where Methiod");
DBList users1 = DBObject.Where("id", "<=", 1313, typeof(DBUser), Utility.connMy);
foreach (DBUser u4 in users1)
{
Console.WriteLine(u4.ID + " : " + u4.FullName);
}

Next step.

I have two thoughts of what the next step should be. The first is the ability to return a collection of objects (maybe LINQ computable, but I'm not worried about that right now.) The other is that the object should be able to fetch it's own tags.

In my system, each object has tags associated with it. This allows objects to 'subscribe' to other objects. For example, we have a user object and a task object. The user can use tags to subscribe to a task (or many tasks). Many users can subscribe to a same task using this method. The tasks also have an optional prefix. This prefix can say things like; who owns the task, who can modify it, who can complete it, etc. (Just having the tag means you've subscribed and can thus view it.)
So when I get the object it really needs to go and fetch all if it's tags as well.

Because objects rarely have only a single tag, I should probably start with the collections. I'm going to call them lists from now on. These lists must inherit from CollectionBase class in order keep compatibility.

So I think what I want, is a DBList : CollectionBase object that can be used as the return value for static methods.

So I could do something like 
(DBList)DBObject.Where("Age", ">=", "25", typeof(DBUser), Utility.connMy);
and it will return a list of all the users whos age is 25 or older. Or
(DBList)DBObject.Where("Nake", "LIKE", "%chris%", typeof(DBUser), Utility.connMy);

I'm not opposed at all of passing the evaluator as text (">=", "LIKE", "=", "<", etc.) Where this does become a problem is when we need to specify more than a single Where clause. But I figure thats a bridge we can cross when we come to it.

Wednesday, February 25, 2009

DBObject rev .03

Alright, now I have a working .By
It actually simplifies things a lot, to the point I'm starting to double think making everything static.

As you can see, you can simplly call something like:

DBUser chris = (DBUser)DBObject.By("firstname", "chris", typeof(DBUser), Utility.connMy);
DBUser bob = (DBUser)DBObject.By("id", "1327", typeof(DBUser), Utility.connMy);


And get back a valid user.

This isn't prefect, for example, what if there are two people named Chris? this would only return an object for the first record returned. Which leads into my next step, returning a list of DBObjects.


///
/// This is the baseclass for DB Object created by Chris Richards
///

[Serializable]
public class DBObject
{
//Holds all the properties in the object
protected Hashtable _properties = new Hashtable();

protected DBObject() { }


///
/// Get the Object By a single property
///

/// The Property you want to get the Object By
/// Value of the Property
/// Type of the object to get (Must inherent from DBOBject)
/// Connection String to use
///
static protected DBObject By(string property, object value, Type who_type, string connection)
{
DBObject obj = null; //Either it will be set to a valid object, or it will remain null

//Find the Column Name for the property requested
string column_name = DBObject.GetColumn(property, who_type);

if (column_name != string.Empty)
{
//Append the WHERE clause
string query = DBObject.SelectQuery(who_type) + " WHERE " + column_name + "=?value";

using (MySqlConnection conn = new MySqlConnection(connection))
{
MySqlCommand cmd = new MySqlCommand(query, conn);
cmd.Parameters.AddWithValue("?value", value);

try
{
conn.Open();
MySqlDataReader reader = cmd.ExecuteReader();

if (reader.Read())
{
obj = (DBObject)DBObject.FromReader(reader, who_type);
}

conn.Close();
}
catch (Exception ex)
{
Console.WriteLine("\n\nERROR:\n" + ex.Message);
}
}

}

//Return what we have
return obj;
}

///
/// Gets the Column name of a Property
///

/// Name of the Property
/// Type of the object to get (Must inherent from DBOBject)
///
static protected string GetColumn(string property, Type who_type)
{
foreach (PropertyInfo who_properity in who_type.GetProperties())
{
//Find the matching property
if (who_properity.Name.ToLower() == property.ToLower())
{
foreach (object attribute in who_properity.GetCustomAttributes(false))
{
//Get the column
if (attribute is DBColumn)
{
return ((DBColumn)attribute).Column;
}
}
}
}

//Couldn't find it
return string.Empty;
}

///
/// Gets the SELECT ... FROM query for this object
///

/// Type of the object to get (Must inherent from DBOBject)
///
static protected string SelectQuery(Type who_type)
{
//Generate the Select Query
string _select = "SELECT";

//Now Set all the select columns
foreach (PropertyInfo properity in who_type.GetProperties())
{
foreach (object attribute in properity.GetCustomAttributes(false))
{
if (attribute is DBColumn)
{
_select += " " + ((DBColumn)attribute).Column + ",";
}
}
}

//Remove the last Comma
_select = _select.Substring(0, _select.Length - 1);

//Get the Table Name for this Object
foreach (object attr in who_type.GetCustomAttributes(false))
{
if (attr is DBTable)
{
//Add the Table to our List
_select += " FROM " + ((DBTable)attr).Table;
}
}
//Now Return it
return _select;
}

///
/// This will populate the object from the reader.
/// It will not advance the reader.
///

///
/// Type of the object to get (Must inherent from DBOBject)
///
static protected DBObject FromReader(MySqlDataReader reader, Type who_type)
{
DBObject who = (DBObject)Activator.CreateInstance(who_type, true);
//Look for all the properities
foreach (PropertyInfo properity in who_type.GetProperties())
{
if (properity.CanWrite)
{
object[] attributes = properity.GetCustomAttributes(false);
foreach (object attribute in attributes)
{
if (attribute is DBColumn)
{
try
{
properity.SetValue(who, reader[((DBColumn)attribute).Column], null);
}
catch (IndexOutOfRangeException)
{
//Just Skip it
}
catch (Exception ex)
{
string junk = ex.Message;
}
}
}
}
}
return who;
}
}


Here is my test object.


[Serializable]
[DB.MetaData.DBTable("users")]
class DBUser : DBObject
{
[DBColumn("first_name")]
public string FirstName
{
get
{
if (!_properties.ContainsKey("first_name"))
{
_properties["first_name"] = string.Empty;
}
return (string)_properties["first_name"];
}

set
{
_properties["first_name"] = value;
}
}

[DBColumn("last_name")]
public string LastName
{
get
{
if (!_properties.ContainsKey("last_name"))
{
_properties["last_name"] = string.Empty;
}
return (string)_properties["last_name"];
}

set
{
_properties["last_name"] = value;
}
}

public string FullName
{
get { return this.FirstName + " " + this.LastName; }
}

[DBColumn("id", true)]
public int ID
{
get
{
if (!_properties.ContainsKey("id"))
{
_properties["id"] = -1;
}
return (int)_properties["id"];
}

set
{
_properties["id"] = value;
}
}


protected DBUser() { }

static public void DEBUG()
{

string suery = DBObject.SelectQuery(typeof(DBUser)) + " WHERE id=?id";

Console.Write("Going to run: " + suery);

using (MySqlConnection conn = new MySqlConnection(Utility.connMy))
{
MySqlCommand cmd = new MySqlCommand(suery, conn);
cmd.Parameters.AddWithValue("?id", 1327);

try
{
conn.Open();
MySqlDataReader reader = cmd.ExecuteReader();

if (reader.Read())
{
DBUser user = (DBUser)DBObject.FromReader(reader, typeof(DBUser));
Console.WriteLine("\nGot: " + user.FullName);
}

conn.Close();
}
catch (Exception ex)
{
Console.WriteLine("\n\nERROR:\n" + ex.Message);
}
}

//Now try the properity
Console.WriteLine("\n\nTrying the By Methiod (ID)");
DBUser u2 = (DBUser)DBObject.By("id", "1327", typeof(DBUser), Utility.connMy);
if (u2 != null)
{
Console.WriteLine("Got: " + u2.FullName);
}
else
{
Console.WriteLine("Oops, got a null back");
}

//Try another property
Console.WriteLine("\n\nTrying the By Methiod (FirstName)");
DBUser u3 = (DBUser)DBObject.By("firstname", "chris", typeof(DBUser), Utility.connMy);
if (u3 != null)
{
Console.WriteLine("Got: " + u3.FullName);
}
else
{
Console.WriteLine("Oops, got a null back");
}
}
}

Next Steps

So far I think things are going good. I could even start to use this in the real world. But I'd like to complete a few more features first.

Having a .By(Property, Value) method. That way we could get an object by any defined property.

Having something like .Get(Property) and .Set(Property) that will return the value of any property. Right now the child class has to access the underlying _properties hash. I don't necessarily like that idea, it gives the developer a chance to screw up more than just the property they are working with (since all properties are through the same hash.) Also methods like that would mean that the developer wouldn't have to check if the property already exists and if they should return a default value. On the down side, how do you define what the default should be? probably in the attribute.

DBObject rev .02

Ok, here is the code with the static modifications.
I like this better, but I don't like the whole "(DBUser)DBObject.FromReader(reader, typeof(DBUser));" (or "(DBUser)DBUser.FromReader(reader, typeof(DBUser));") thing. It just seems like I'm having to type to much. Then again, the child class could encapsulate some of this passing by type.

DBObject



///
/// This is the baseclass for DB Object created by Chris Richards
///

[Serializable]
public class DBObject
{
//Holds all the properties in the object
protected Hashtable _properties = new Hashtable();

protected DBObject() { }

///
/// This will populate all of the properties from the table by the column 'id'
///

///
///
public object ByID(int id)
{
return new DBObject();
}

///
/// Gets the SELECT ... FROM query for this object
///

static protected string SelectQuery(Type who)
{
//Generate the Select Query
string _select = "SELECT";

//Now Set all the select columns
foreach (PropertyInfo properity in who.GetProperties())
{
foreach (object attribute in properity.GetCustomAttributes(false))
{
if (attribute is DBColumn)
{
_select += " " + ((DBColumn)attribute).Column + ",";
}
}
}

//Remove the last Comma
_select = _select.Substring(0, _select.Length - 1);

//Get the Table Name for this Object
foreach (object attr in who.GetCustomAttributes(false))
{
if (attr is DBTable)
{
//Add the Table to our List
_select += " FROM " + ((DBTable)attr).Table;
}
}
//Now Return it
return _select;
}

///
/// This will populate the object from the reader.
/// It will not advance the reader.
///

///
///
static protected DBObject FromReader(MySqlDataReader reader, Type who_type)
{
DBObject who = (DBObject)Activator.CreateInstance(who_type, true);
//Look for all the properities
foreach (PropertyInfo properity in who_type.GetProperties())
{
if (properity.CanWrite)
{
object[] attributes = properity.GetCustomAttributes(false);
foreach (object attribute in attributes)
{
if (attribute is DBColumn)
{
try
{
properity.SetValue(who, reader[((DBColumn)attribute).Column], null);
}
catch (IndexOutOfRangeException)
{
//Just Skip it
}
catch (Exception ex)
{
//ErrorLog.Log(-1, this.GetType().Name + " reader Error", ex.Message);
string junk = ex.Message;
}
}
}
}
else
{
ErrorLog.Log(-1, "Can't Write " + who.GetType().Name + "'s Properity", "Properity: " + properity.Name);
}
}
return who;
}
}


Test Code


And here is the test example

string suery = DBObject.SelectQuery(typeof(DBUser)) + " WHERE id=?id";

Console.Write("Going to run: " + suery);

using (MySqlConnection conn = new MySqlConnection(Utility.connMy))
{
MySqlCommand cmd = new MySqlCommand(suery, conn);
cmd.Parameters.AddWithValue("?id", 1327);

try
{
conn.Open();
MySqlDataReader reader = cmd.ExecuteReader();

if (reader.Read())
{
DBUser user = (DBUser)DBObject.FromReader(reader, typeof(DBUser));
Console.WriteLine("\nGot: " + user.FirstName);
}

conn.Close();
}
catch (Exception ex)
{
Console.WriteLine("\n\nERROR:\n" + ex.Message);
}
}

Static again

I haven't been able to work on this for a while because of school and work. I was rereading my last post and I noticed that I have to create a new DBUser in order to get the SelectQuery for it. I don't like this, it violates my 'new' rule. One of the reasons it's not a static method is because it caches the query string. But it doesn't really matter if it caches the query if you call a new object each time, it eliminates the point of the cache.

So this is where I'm going to pick things back up, I'm going to make those two static and post the code.

The only problem is, how so I make these static? The base needs to know the child's type to work. When they wheren't static I could just use the this keyword. But as a static methiod I no longer have access to that keyword. The only alternative I can think of off the top of my head is requireing a type passed to both methods. I don't know that I quite like that idea.

Thursday, January 1, 2009

Decisions

So as I'm thinking about the DBOBject I just created. I ask myself, "Why make a protected method FromReader() vs just using a constructor? Why a protected method and not just a static method?"

The way I see it, using the "new" keyword to create a database object should create a table in the database. So using the datareader in the constructor isn't an option because we are retrieving the object from the database, not adding one to it.

The static question is a bit tricker. I'm not 100% sure that it shouldn't be a a static method. It would eliminate a step. right now you have to do something like:


DBUser user = new DBUser();

string suery = user.SelectQuery + " WHERE id=?id";

Console.Write("Going to run: " + suery);

using (MySqlConnection conn = new MySqlConnection(Utility.connMy))
{
MySqlCommand cmd = new MySqlCommand(suery, conn);
cmd.Parameters.AddWithValue("?id", 1327);

try
{
conn.Open();
MySqlDataReader reader = cmd.ExecuteReader();

if (reader.Read())
{
user.FromReader(reader);
}

conn.Close();
}
catch (Exception ex)
{
Console.WriteLine("\n\nERROR:\n" + ex.Message);
}
}



Which means we are still calling new on the object (even though it's internally, it still violates the whole 'new should be to add a record' idea. If I changed .FromReader to be a static method, then we could simply do this:


DBUser user = null;

string suery = user.SelectQuery + " WHERE id=?id";

Console.Write("Going to run: " + suery);

using (MySqlConnection conn = new MySqlConnection(Utility.connMy))
{
MySqlCommand cmd = new MySqlCommand(suery, conn);
cmd.Parameters.AddWithValue("?id", 1327);

try
{
conn.Open();
MySqlDataReader reader = cmd.ExecuteReader();

if (reader.Read())
{
user = DBUser.FromReader(reader);
}

conn.Close();
}
catch (Exception ex)
{
Console.WriteLine("\n\nERROR:\n" + ex.Message);
}
}


It still takes the same number of lines of code, but it doesn't violate the "new keyword" rule.

I think I like the method being static. Mostly because it stays constant.