d2jsp
Log InRegister
d2jsp Forums > Off-Topic > Computers & IT > Programming & Development > A Pet Project
12Next
Add Reply New Topic New Poll
Member
Posts: 23,862
Joined: Aug 16 2006
Gold: 20.00
Apr 2 2015 10:38pm
VB.NET

Something I've been working on.

We needed something like this at work, but I haven't really had the time to work on it.


I also wanted to do something at home, but just couldn't get a decent web app started.

I love console apps though.




So what we needed basically is a SQL Generator, with decent syntax. My boss wanted it to feel like StringBuilder, so that is what I use internally.

Here is a sample: (very skeleton atm, only like 300 lines deep)

Code
Module Module1

Sub Main()
Dim SQLGenerator As New SQLGenerator()
Dim GeneratedSQL As String

Dim ColumnNames As New List(Of String)

ColumnNames.Add("Column1")
ColumnNames.Add("Column2")
ColumnNames.Add("Column3")

Try
With SQLGenerator
.Select(ColumnNames)
.From("MyTable")
.Where("Column1").Equal(25, True)
.And("Column2").DoesNotEqual(99, False)
.And("Column3").GreaterThan(9001, False)
End With
Catch ex As Exception
Console.WriteLine(ex.ToString())
Return
End Try

GeneratedSQL = SQLGenerator.GetGeneratedSQL()

Dim file As System.IO.StreamWriter
file = My.Computer.FileSystem.OpenTextFileWriter("C:SomePath\GeneratedSql.txt", False)

If Helpers.IsStringValid(GeneratedSQL) Then
file.Write(GeneratedSQL)
End If

file.Close()

End Sub

End Module


########################################################
######################OUTPUT############################

SELECT Column1, Column2, Column3 FROM MyTable WHERE Column1 = '25' AND Column2 <> 99 AND Column3 > 9001


This post was edited by Eep on Apr 2 2015 10:39pm
Member
Posts: 3,386
Joined: May 4 2013
Gold: 1,780.00
Apr 3 2015 03:11am
using ORM is very nice, using raw sql generated by it isn't, because you lose all the benefits and now you have blob of sql that you can't easily edit, and if you do, you're again required to manually sanitize all user input etc
Member
Posts: 23,862
Joined: Aug 16 2006
Gold: 20.00
Apr 3 2015 10:48am
Quote (nuvo @ Apr 3 2015 04:11am)
using ORM is very nice, using raw sql generated by it isn't, because you lose all the benefits and now you have blob of sql that you can't easily edit, and if you do, you're again required to manually sanitize all user input etc


I think I forgot to mention something -

The RAW SQL generated by this will NOT be used directly (as a string command into a DB query) - it will be processed (by already existing code)

Also, this generator is limited to "Select..." statements - no updating/deleting/inserting.

The demo above is literally 1 hr of tinkering just to show the syntax I wanted.

This post was edited by Eep on Apr 3 2015 10:53am
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Apr 3 2015 01:22pm
if it's for personal use, it's one thing, but trying to handle it yourself for production isn't the greatest idea.

my suggestion is to look into other ORMs.

LINQ, Entity Framework, and nHibernate are worth checking out.

here's a snippet in LINQ:
Code
Dim companyNameQuery = _
From cust In nw.Customers _
Where cust.City = "London" _
Select cust.CompanyName


here's a snippet in nHibernate:
Code
public ICollection<Product> GetByCategory(string category)
{
using (ISession session = NHibernateHelper.OpenSession())
{
var products = session
.CreateCriteria(typeof(Product))
.Add(Restrictions.Eq("Category", category))
.List<Product>();
return products;
}
}


odds are you'll just ignore all that. but if you somehow made it down this far, iirc it's called the "builder" design pattern. if you just want basic support, it shouldn't be too difficult to do yourself. just keep subselects and joins in mind.

This post was edited by carteblanche on Apr 3 2015 01:22pm
Member
Posts: 23,862
Joined: Aug 16 2006
Gold: 20.00
Apr 3 2015 01:33pm
Quote (carteblanche @ Apr 3 2015 02:22pm)
if it's for personal use, it's one thing, but trying to handle it yourself for production isn't the greatest idea.

my suggestion is to look into other ORMs.

LINQ, Entity Framework, and nHibernate are worth checking out.

here's a snippet in LINQ:
Code
Dim companyNameQuery = _
From cust In nw.Customers _
Where cust.City = "London" _
Select cust.CompanyName


here's a snippet in nHibernate:
Code
public ICollection<Product> GetByCategory(string category)
{
using (ISession session = NHibernateHelper.OpenSession())
{
var products = session
.CreateCriteria(typeof(Product))
.Add(Restrictions.Eq("Category", category))
.List<Product>();
return products;
}
}


odds are you'll just ignore all that. but if you somehow made it down this far, iirc it's called the "builder" design pattern. if you just want basic support, it shouldn't be too difficult to do yourself. just keep subselects and joins in mind.


Yeah, I am keeping all of that in mind.


Basically, at work, we have something similar to the one I am working on. Problem is, it doesn't support anything beyond "Select [xxx] from [yyy] where [q = p, t = v]"

All I need to do is generate a SQL string. Paramterizing strings and sanitizing them can be done, but it won't be on my end. I pass the string to something else which handles the rest.

I don't know if this counts as an ORM or not. What I imagined is just a more specific type of string builder.

This post was edited by Eep on Apr 3 2015 01:34pm
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Apr 3 2015 01:43pm
Quote (Eep @ Apr 3 2015 03:33pm)
Yeah, I am keeping all of that in mind.


Basically, at work, we have something similar to the one I am working on. Problem is, it doesn't support anything beyond "Select [xxx] from [yyy] where [q = p, t = v]"

All I need to do is generate a SQL string. Paramterizing strings and sanitizing them can be done, but it won't be on my end. I pass the string to something else which handles the rest.

I don't know if this counts as an ORM or not.


i think you skipped a lesson somewhere. sanitation should ideally occur prior to building the query.

can you explain the point of making it? i'm guessing you dont need something DB agnostic. and you're still using string column names, so you're not worried about changes to the table.

if you're not using a real ORM, the only thing i really wanted is a shortcut to the INSERT statement. i really hate how sql uses the parallel array concept to do an insert.

eg: insert into mytable (col1, col2, col3) values (@col1, @col2, @col3)

i'd prefer they be right next to each other. something like:

eg: insert into mytable (col1=@col1, col2=@col2, col3=@col3)

because in my code, we have tables with 100+ columns and some of the developers wrote it out in a huge unmanageable string. and when we wanna add/remove columns, it's a mess. and you see devs writing casts and shit all inside that mess of a sql statement. so having a utility that accepts a list of columns and generates an insert statement (casting, validating as you go, etc) is nice. but for Select? way too much going on there.

This post was edited by carteblanche on Apr 3 2015 01:52pm
Member
Posts: 23,862
Joined: Aug 16 2006
Gold: 20.00
Apr 3 2015 02:18pm
Quote (carteblanche @ Apr 3 2015 02:43pm)
i think you skipped a lesson somewhere. sanitation should ideally occur prior to building the query.

can you explain the point of making it? i'm guessing you dont need something DB agnostic. and you're still using string column names, so you're not worried about changes to the table.

if you're not using a real ORM, the only thing i really wanted is a shortcut to the INSERT statement. i really hate how sql uses the parallel array concept to do an insert.

eg: insert into mytable (col1, col2, col3) values (@col1, @col2, @col3)

i'd prefer they be right next to each other. something like:

eg: insert into mytable (col1=@col1, col2=@col2, col3=@col3)

because in my code, we have tables with 100+ columns and some of the developers wrote it out in a huge unmanageable string. and when we wanna add/remove columns, it's a mess. and you see devs writing casts and shit all inside that mess of a sql statement. so having a utility that accepts a list of columns and generates an insert statement (casting, validating as you go, etc) is nice. but for Select? way too much going on there.




Well, maybe I've been explaining this badly?

At work, we do something like:

Code
Dim myString as String
Dim collectionOfParameters as Dictionary(Of String, Object)
Dim myResponse as DataTable

'This essentially creates a string like: "Select ColumnName from Table Where ColumnName = @ColumnName"
myString = ClassWhichRepresentsATable.CreateSQLStatement(ByVal SelectColumnNames as List(Of String), ByVal WhereColumnNames as List(of String))

collectionOfParameters = New Dictionary(Of String, Object)
collectionOfParameters.Add(MyWhereColumnNameAsParameter, myValueIWantToPass)

myResponse = OurDAO.GetDataTable(myString, collectionOfParameters)


The thing I am working on is purely for DATA RETRIEVAL. There will be NO insert/delete/update commands attached to it. That is ALL handled somewhere else in our architecture.

Lots of times we have to write one-off methods to retrieve something from a database table. There is currently no good way to retrieve by complex queries, other than hard-coding SQL strings (which is the most readable, but its a hardcoded string...), the method I just typed above, or a bad mixture of the two (where they try to use hard-coded strings to make up for the parts the existing sql generator is missing)

This post was edited by Eep on Apr 3 2015 02:20pm
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Apr 3 2015 03:54pm
company i worked for 6 years ago did something similar years prior to that since ORMs werent readily on the market. their Entity had a map for all their columns, and using inheritance could build its general select statement with joins. then they had the concept of a CriteriaSet which did similar to what you want to do. it essentially just tacked on an extra piece on the end of the where clause.

say Product inherits from SecuredEntity (which adds a join to a security table and includes a condition in the where clause) which inherits from Entity (which handles the base data access code)

eg:
Code
CriteriaSet cs = new CriteriaSet();
cs.Add("Quantity", CriteriaType.EQUAL, userinput);
cs.Add("Id", CriteraType.IN_SQL, "select id from cart");
cs.Limit = 10;

List<Product> products = new Product().find(cs);


so a find sql without criteria would look similar to this (using the mapping and superclasses)
Code
select id, name, quantity, code, ....
from products p
join securitytable st on st.entity_guid = p.guid
where 1=1
and st.user_oid=1
and st.access > 0
limit 2147483647


then using the criteria set just did this:
Quote
select id, name, quantity, code, ....
from products p
join securitytable st on st.entity_guid = p.guid
where 1=1
and st.user_oid=1
and st.access > 0
and ((quantity = 4) and (id in (select id from cart)))
limit 10


they decided early on they weren't building an entire engine. they just wanted a tool to add extra stuff in the where clause. and they'd just nest CriteriaSets as needed. maybe it'll give you some ideas?

Quote
There is currently no good way to retrieve by complex queries


i just mentioned the insert because flat insert...values is very easy. SELECT is much more difficult. and when you start trying to write something that will handle your use cases, you'll quickly add more complexity.

eg:
select a, b, c, (select * from blah join boohoo ...) from (select * from nastytable union select * from nicetable) where some_id in (select id from another_table) and (some_id not in (select id from crap_table join asdf ....) or new date().add(3, 'd') < other_date) and not exists (select 1 from ....)

now the code you write via your sqlbuilder will be less readable than the sql. and you don't want to have this sql first, then sanitize it. if your unit tests aren't well maintained, then you'll be in for a nasty surprise when you add an extra feature to support something you want and it impacts existing code quietly.

Quote
other than hard-coding SQL strings (which is the most readable, but its a hardcoded string...)

i suggest just writing stored procedures then passing params.
Member
Posts: 23,862
Joined: Aug 16 2006
Gold: 20.00
Apr 3 2015 04:08pm
Quote (carteblanche @ Apr 3 2015 04:54pm)
company i worked for 6 years ago did something similar years prior to that since ORMs werent readily on the market. their Entity had a map for all their columns, and using inheritance could build its general select statement with joins. then they had the concept of a CriteriaSet which did similar to what you want to do. it essentially just tacked on an extra piece on the end of the where clause.

say Product inherits from SecuredEntity (which adds a join to a security table and includes a condition in the where clause) which inherits from Entity (which handles the base data access code)

eg:
Code
CriteriaSet cs = new CriteriaSet();
cs.Add("Quantity", CriteriaType.EQUAL, userinput);
cs.Add("Id", CriteraType.IN_SQL, "select id from cart");
cs.Limit = 10;

List<Product> products = new Product().find(cs);


so a find sql without criteria would look similar to this (using the mapping and superclasses)
Code
select id, name, quantity, code, ....
from products p
join securitytable st on st.entity_guid = p.guid
where 1=1
and st.user_oid=1
and st.access > 0
limit 2147483647


then using the criteria set just did this:


they decided early on they weren't building an entire engine. they just wanted a tool to add extra stuff in the where clause. and they'd just nest CriteriaSets as needed. maybe it'll give you some ideas?



i just mentioned the insert because flat insert...values is very easy. SELECT is much more difficult. and when you start trying to write something that will handle your use cases, you'll quickly add more complexity.

eg:
select a, b, c, (select * from blah join boohoo ...) from (select * from nastytable union select * from nicetable) where some_id in (select id from another_table) and (some_id not in (select id from crap_table join asdf ....) or new date().add(3, 'd') < other_date) and not exists (select 1 from ....)

now the code you write via your sqlbuilder will be less readable than the sql. and you don't want to have this sql first, then sanitize it. if your unit tests aren't well maintained, then you'll be in for a nasty surprise when you add an extra feature to support something you want and it impacts existing code quietly.


i suggest just writing stored procedures then passing params.


I see what you mean.

It really can become insanely complex real quickly depending on what developers want/expect out of it....

and at the end of the day, if a complex SQL query looks worse than a hard-coded string, what was the point?


I just wanted something to do that wasn't web involved. I can write for hours on console apps. I just can't seem to find anything interesting to do for a web app, and I also tend to get lost in the big picture.

This post was edited by Eep on Apr 3 2015 04:10pm
Member
Posts: 32,925
Joined: Jul 23 2006
Gold: 3,804.50
Apr 3 2015 04:33pm
Quote (Eep @ Apr 3 2015 06:08pm)
I just wanted something to do that wasn't web involved. I can write for hours on console apps. I just can't seem to find anything interesting to do for a web app, and I also tend to get lost in the big picture.


wanna work on my todo list?

i'm using titanium appcelerator at work. the mvc model requires certain convention.

eg:
controller file goes in app/controllers/name.js
view file goes in app/views/name.xml

to refer to it in code:
require('name');

unfortunately, moving/renaming it isn't easy. 1) i have to move both files separately. IDE won't do it for me for whatever reason. 2) all my require statements have to be updated to include the path (after app/controllers)

eg:
controller file goes in app/controllers/inventory/name.js
view file goes in app/views/inventory/name.xml

to refer to it in code:
require('inventory/name');


wanna write me a plugin for titanium appcelerator (aptana or eclipse based) so i can right click -> refactor -> move/rename and it handles all that for me? i've got a python script i call from the terminal to move files, and i used a wrapper around require(..) to get around it. but it would be handy. i've never written a plugin before, but hey i'm sure you can do it.

This post was edited by carteblanche on Apr 3 2015 04:35pm
Go Back To Programming & Development Topic List
12Next
Add Reply New Topic New Poll