Never Say Never » Tally Table

Tally Table CTE

Seth Phelabaum — Fri, 19 Mar 2010 13:24:48 +0000

Now that I have several posts on what you can do with a Tally table, I figured I’d share my favorite way to create one inline. I still prefer to have a physical tally table (usually in a Utility database that can be accessed from anywhere and doesn’t need to be created in each individual database) for permament code, but for times when you need one on the fly, this is my preferred method. I can’t really take the credit for this query, the base construct is based on something I’ve seen attributed to Itzik Ben-Gan. I’ve modified it a bit and changed up the formatting to be the way I like it. Anything over a few thousand rows I’d probably use a physical tally table for, but on small numbers you shouldn’t see much of a performance hit with this script.

-- Tally Table CTE script (SQL 2005+ only)
-- You can use this to create many different numbers of rows... for example:
-- You could use a 3 way cross join (t3 x, t3 y, t3 z) instead of just 2 way to generate a different number of rows.
-- The # of rows this would generate for each is noted in the X3 comment column below.
-- For most common usage, I find t3 or t4 to be enough, so that is what is coded here.
-- If you use t3 in ‘Tally’, you can delete t4 and t5.

; WITH
-- Tally table Gen            Tally Rows:     X2                X3
t1 AS (SELECT 1 N UNION ALL SELECT 1 N),    -- 4            ,    8
t2 AS (SELECT 1 N FROM t1 x, t1 y),            -- 16            ,    64
t3 AS (SELECT 1 N FROM t2 x, t2 y),            -- 256            ,    4096
t4 AS (SELECT 1 N FROM t3 x, t3 y),            -- 65536        ,    16,777,216
t5 AS (SELECT 1 N FROM t4 x, t4 y),            -- 4,294,967,296,    A lot
Tally AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) N
FROM t3 x, t3 y) -- Change the t3's to one of the other numbers above for more/less rows

Tally Table – Delimited list to Table

Seth Phelabaum — Sat, 27 Feb 2010 20:49:27 +0000

Dealing with delimited lists (Usually separated by a comma) in SQL is a problem easily handled by a simple function and a Tally Table. (Tally tables are also often referred to as Numbers tables or spt_values tables. If you still don’t know what that is, please see this excellent article on Tally tables written by my friend and SSC heavyweight Jeff Moden.) This particular implementation is somewhat specific in nature but can give you an alternative to Dynamic SQL when you want to pass in a list as a parameter and do an IN in a Stored Procedure. The following function will take your delimiter and string and parse it into a table so you can do your IN. (I’m leaving my standard header on the function in this case because there are some good notes in there.)

/*
=============================================================================================
CREATE DATE:     02/27/2010
LAST MODIFIED:    02/27/2010
CREATED BY:        SETH PHELABAUM
PURPOSE:        Splits a string based on a passed in delimiter and returns a table.
ISSUES:            Strings with extra 's will break this function, handle that on the end that calls it.
Notes:            To make it a simpler function, I removed the peice that trimmed spaces around commas. Do
this before or after calling it.
Revision History:
Date     By        Change Made
-------- ---      -------------------------------------
=============================================================================================
GRANT SELECT ON TVF_TallySplit TO [Somebody]
SELECT * FROM TVF_TallySplit(',','Orange,Apple,Banana,Pear,Watermelon,Grape')
SELECT * FROM TVF_TallySplit('*','Orange*Apple*Banana*Pear*Watermelon*Grape')
DROP FUNCTION TVF_TallySplit
*/

CREATE FUNCTION TVF_TallySplit(
@Delim            CHAR(1),            -- List Delimiter
@String            VARCHAR(8000))
RETURNS TABLE
AS

RETURN(
SELECT SUBSTRING(@Delim + @String + @Delim,N+1,CHARINDEX(@Delim,@Delim + @String + @Delim,N+1)-N-1) ListValue
FROM Tally
WHERE N < LEN(@Delim + @String + @Delim)
AND SUBSTRING(@Delim + @String + @Delim,N,1) = @Delim )

What to do with this
Let’s say you have a table containing names of your favorite fruits. (In case you were wondering… No, these aren’t my favorite fruits; they were just ones that immediately came to mind when writing this. I don’t even like half of these.)

CREATE TABLE Fruits(
Name VARCHAR(25))

INSERT INTO Fruits(Name)
SELECT 'Apple' UNION ALL SELECT 'Banana' UNION ALL SELECT 'Grapefruit' UNION ALL
SELECT 'Kiwi' UNION ALL SELECT'Tomatoe' UNION ALL SELECT 'Grape' UNION ALL
SELECT 'Orange' UNION ALL SELECT 'DragonFruit' UNION ALL SELECT 'Strawberry'

You then ask someone else what their favorite fruits are and want to see what fruits you have in common. You might think you could just write a query for that like this:

DECLARE @YFFruits VARCHAR(200)

SET @YFFruits = 'Orange,Apple,Banana,Pear,Watermelon,Grape'
-- OR SET @YFFruits = '''Orange'',''Apple'',''Banana'',''Pear'',''Watermelon'',''Grape'''

SELECT * FROM Fruits WHERE Name LIKE (@YFFruits)
-- OR SELECT * FROM Fruits where Name IN (@YFFruits)

However, this won’t work because in all these cases SQL is looking for a single fruit named : ‘Orange,Apple,Banana,Pear,Watermelon,Grape’ not any fruit in what is really a list.

A common solution for this is to use Dynamic SQL which would make your query this:

DECLARE @YFFruits VARCHAR(200)
SET @YFFruits = '''Orange'',''Apple'',''Banana'',''Pear'',''Watermelon'',''Grape'''
EXEC('SELECT * FROM Fruits WHERE Name IN (' + @YFFruits + ')')

This works and will properly match up your fruits.

The above function allows you to accomplish your goal without Dynamic SQL with a query that looks like this:

DECLARE @YFFruits VARCHAR(200)
SET @YFFruits = 'Orange,Apple,Banana,Pear,Watermelon,Grape'
SELECT * FROM Fruits WHERE Name IN (SELECT * FROM Util.dbo.TVF_TallySplit(',',@YFFruits))

You can also simply join to the table instead of using the IN keyword which gives you more flexibility in your query writing.

DECLARE @YFFruits VARCHAR(200)
SET @YFFruits = 'Orange,Apple,Banana,Pear,Watermelon,Grape'
SELECT Fruits.*
FROM Fruits
INNER JOIN Util.dbo.TVF_TallySplit(',',@YFFruits)) T ON Fruits.Name = T.ListValue

Note that because I wanted to keep the function somewhat simple, it does not handle extra spaces around the commas. Single quotes within the string will also break it which limits its usage somewhat. If this is a concern for your implementation, you either need to replace the single quotes on both sides or use a different method. Despite the fact that the example above uses a list of strings, in real life situations I use this mainly for lists of uniqueidentifiers or numbers where single quotes/spaces are never an issue.

Tally Table – String Cleaning

Seth Phelabaum — Wed, 13 Jan 2010 14:37:11 +0000

Overview
This is something that most people eventually need for reporting purposes. This function uses a Tally table to ‘clean’ a string, removing anything you don’t specify in the @Rep parameter. In the case below, I remove everything but letters, numbers, spaces, commas and periods. This is a slightly modified version of a function I wrote for something else, so the specifics like replacing a lot of the double spaces, and replacing carriage returns and line breaks with spaces were specifically for that. Remove those pieces of code if you wish, or add those as additional parameters to the function.

Other Methods
I’ll mention that I’ve seen people claim a while loop will beat this method and others that say this method is still faster. I haven’t tested this enough personally to make the claim one way or another, but I prefer this one. This is also one of the accepted places where a CLR function can be superior to T-SQL, but for people who can’t or don’t want to use CLR on their servers, this will still work.

Tally Method

/*
SELECT dbo.SSC_fn_TallyClean('This will remove all of this---->!)#%*)^+__#@#$+_!)#~!!! <----.','[a-zA-Z0-9. ,]') Cleaned
*/

CREATE FUNCTION SSC_fn_TallyClean(
@A VARCHAR(500),
@Rep VARCHAR(100))
RETURNS VARCHAR(500)

AS
BEGIN
DECLARE @B VARCHAR(500)
SET @B = '' -- Initialize @B

-- Remove Line Feed / Carriage Returns (The tally code would have removed them, but I wanted to replace them with spaces for readability.)
SET @A = REPLACE(REPLACE(@A,CHAR(10),' '),CHAR(13),' ')

SELECT @B = @B + SUBSTRING(@A,N,1)
FROM Tally
WHERE N <= DATALENGTH(@A)
AND SUBSTRING(@A,N,1) LIKE @Rep -- Remove everything but letters, numbers, spaces, period and comma.

RETURN REPLACE(REPLACE(@B,' ',' '),' ',' ') -- Removes some double spaces.
END

Tally Table – Character Date Validation

Seth Phelabaum — Tue, 15 Dec 2009 00:34:48 +0000

Tally (or numbers) tables are one of my favorite query writing tools. Such a simple premise that can be applied to a wide variety of problems. If you don’t know what they are, I’d recommend this article. Here is one problem I’ve seen a few times now that you can use a Tally table to easily solve. This is by no means the only way to do something like this, nor is it necessarily even the *best* way, but it is *a* way.

The Problem:

You have a char/varchar column that holds dates in YYYYMMDD format. You now need to cast these values to dates. The problem is, with no validation on the column, some of these values aren’t actually valid dates and you get conversion errors. You’d like to go through and identify anything that didn’t follow the pattern of YYYYMMDD or wasn’t a valid date so that it can be either fixed or removed.

The Tally Solution:

The following query uses a Tally table to generate a range of valid dates that you can check your varchar field against. The below version works in 2000+. (You can just as easily use a CTE tally in 2005/2008)

DECLARE @StartDate datetime
DECLARE @EndDate datetime

SET @StartDate = '19910101 00:00:00'
SET @EndDate = '20250101 00:00:00'

SELECT *
FROM YourTable
WHERE vcdatecolumn NOT IN (
SELECT CAST(YEAR(dateadd(d,n,@StartDate)) AS CHAR(4)) +
RIGHT('0' + CAST(MONTH(dateadd(d,n,@StartDate)) AS VARCHAR(2)),2) +
RIGHT('0' + CAST(DAY(dateadd(d,n,@StartDate)) AS VARCHAR(2)),2)
FROM Tally
WHERE DATEADD(d,n,@StartDate) <= @EndDate
)

Other Methods:
As mentioned, there are a lot of ways to do this. Because ISDATE() jumps out as being such an easy one, I’ll mention a couple of the problems with using that method. The main gap left is that some of the junk values (fairly common with this setup) like ’9901′ are valid dates according to ISDATE(). However, it is unlikely that this date is really supposed to be 9901/01/01. You can add a length check, but then you run into ’9901June’ still being valid, etc. etc. Using the Tally method, all of these are simply flagged as invalid and you can make up your mind about what to do with them once they’ve been identified.