Never Say Never » String Manipulation

Tally Table – String Cleaning

Seth Phelabaum — Wed, 13 Jan 2010 14:37:11 +0000

Overview
This is something that most people eventually need for reporting purposes. This function uses a Tally table to ‘clean’ a string, removing anything you don’t specify in the @Rep parameter. In the case below, I remove everything but letters, numbers, spaces, commas and periods. This is a slightly modified version of a function I wrote for something else, so the specifics like replacing a lot of the double spaces, and replacing carriage returns and line breaks with spaces were specifically for that. Remove those pieces of code if you wish, or add those as additional parameters to the function.

Other Methods
I’ll mention that I’ve seen people claim a while loop will beat this method and others that say this method is still faster. I haven’t tested this enough personally to make the claim one way or another, but I prefer this one. This is also one of the accepted places where a CLR function can be superior to T-SQL, but for people who can’t or don’t want to use CLR on their servers, this will still work.

Tally Method

/*
SELECT dbo.SSC_fn_TallyClean('This will remove all of this---->!)#%*)^+__#@#$+_!)#~!!! <----.','[a-zA-Z0-9. ,]') Cleaned
*/

CREATE FUNCTION SSC_fn_TallyClean(
@A VARCHAR(500),
@Rep VARCHAR(100))
RETURNS VARCHAR(500)

AS
BEGIN
DECLARE @B VARCHAR(500)
SET @B = '' -- Initialize @B

-- Remove Line Feed / Carriage Returns (The tally code would have removed them, but I wanted to replace them with spaces for readability.)
SET @A = REPLACE(REPLACE(@A,CHAR(10),' '),CHAR(13),' ')

SELECT @B = @B + SUBSTRING(@A,N,1)
FROM Tally
WHERE N <= DATALENGTH(@A)
AND SUBSTRING(@A,N,1) LIKE @Rep -- Remove everything but letters, numbers, spaces, period and comma.

RETURN REPLACE(REPLACE(@B,' ',' '),' ',' ') -- Removes some double spaces.
END

String Manipulation – CHARINDEX()

Seth Phelabaum — Sat, 19 Dec 2009 18:34:22 +0000

I see charindex used quite commonly in string manipulation. What I rarely see used is the optional third parameter. Here I will explain how to use it, a situation where it can be useful and something to be careful of.

FROM BOL:
Syntax
CHARINDEX ( expression1 ,expression2 [ , start_location ] )

Arguments
expression1
Is a character expression that contains the sequence to be found. expression1 is limited to 8000 characters.

expression2
Is a character expression to be searched.

start_location
Is an integer or bigint expression at which the search starts. If start_location is not specified, is a negative number, or is 0, the search starts at the beginning of expression2.

How it works
Let’s say we have a string with a set of characters in it and you want to find what is in between them. Easy enough if they’re different characters like [] or () (You could still use this for that situation to ensure you didn’t get an ending char before your starting char, and the warning below is still valid). But what if they’re the same character such as "" or –? Here’s how to use the third parameter of charindex along with a substring to accomplish this.

-- Create a sample string with a pair of quotes.  The goal is to get what is in between the quotes. 
declare @a varchar(50)
SET @a = 'This is a "test" string.' 

-- Find the First " in the string 
SELECT CHARINDEX('"',@a)
-- Find the Second " in the string by starting one character past the first one 
SELECT CHARINDEX('"',@a,CHARINDEX('"',@a)+1)
-- Put it together 
SELECT SUBSTRING(@a,
                -- Start from first " 
                CHARINDEX('"',@a),
                -- Position of the second " minus the position of the first " to find length. 
                CHARINDEX('"',@a,CHARINDEX('"',@a)+1) - CHARINDEX('"',@a))

We’re almost there. If you run the code, you’ll notice that the result is "test . Let’s trim off that extra ".

-- Clean it up 
SELECT SUBSTRING(@a,
                -- Add 1 to trim off beginning quote 
                CHARINDEX('"',@a)+1,
                -- Subtract 1 to trim off ending quote                            
                CHARINDEX('"',@a,CHARINDEX('"',@a)+1)-1 - CHARINDEX('"',@a))

Caution

This is really a caution of the substring function, not charindex, but it is something that will cause the above technique to fail (and error). The problem occurs when you do not find the characters that you’re looking for. In the above example, I was looking for two "’s. If they don’t exist, the code will error out. If no quotes are present, it works up until the cleanup portion. If only one " is present, it fails at the first substring. You will receive this error:

Msg 537, Level 16, State 2, Line 20

Invalid length parameter passed to the LEFT or SUBSTRING function.

The reason for this is that it attempts to pass a negative length to the substring function, which isn’t allowed. To safeguard against this, you can use a case statement like the following:

DECLARE @a varchar(50)
SET @a = 'This is a "fail  string.' 

                -- Test for a second " before attempting the substring. 
SELECT CASE WHEN CHARINDEX('"',@a,CHARINDEX('"',@a)+1)-1 - CHARINDEX('"',@a)) > 0 THEN
        SUBSTRING(@a,
                CHARINDEX('"',@a)+1,
                CHARINDEX('"',@a,CHARINDEX('"',@a)+1)-1 - CHARINDEX('"',@a))
            ELSE 'Not Found'
            End

Tally Table – Character Date Validation

Seth Phelabaum — Tue, 15 Dec 2009 00:34:48 +0000

Tally (or numbers) tables are one of my favorite query writing tools. Such a simple premise that can be applied to a wide variety of problems. If you don’t know what they are, I’d recommend this article. Here is one problem I’ve seen a few times now that you can use a Tally table to easily solve. This is by no means the only way to do something like this, nor is it necessarily even the *best* way, but it is *a* way.

The Problem:

You have a char/varchar column that holds dates in YYYYMMDD format. You now need to cast these values to dates. The problem is, with no validation on the column, some of these values aren’t actually valid dates and you get conversion errors. You’d like to go through and identify anything that didn’t follow the pattern of YYYYMMDD or wasn’t a valid date so that it can be either fixed or removed.

The Tally Solution:

The following query uses a Tally table to generate a range of valid dates that you can check your varchar field against. The below version works in 2000+. (You can just as easily use a CTE tally in 2005/2008)

DECLARE @StartDate datetime
DECLARE @EndDate datetime

SET @StartDate = '19910101 00:00:00'
SET @EndDate = '20250101 00:00:00'

SELECT *
FROM YourTable
WHERE vcdatecolumn NOT IN (
SELECT CAST(YEAR(dateadd(d,n,@StartDate)) AS CHAR(4)) +
RIGHT('0' + CAST(MONTH(dateadd(d,n,@StartDate)) AS VARCHAR(2)),2) +
RIGHT('0' + CAST(DAY(dateadd(d,n,@StartDate)) AS VARCHAR(2)),2)
FROM Tally
WHERE DATEADD(d,n,@StartDate) <= @EndDate
)

Other Methods:
As mentioned, there are a lot of ways to do this. Because ISDATE() jumps out as being such an easy one, I’ll mention a couple of the problems with using that method. The main gap left is that some of the junk values (fairly common with this setup) like ’9901′ are valid dates according to ISDATE(). However, it is unlikely that this date is really supposed to be 9901/01/01. You can add a length check, but then you run into ’9901June’ still being valid, etc. etc. Using the Tally method, all of these are simply flagged as invalid and you can make up your mind about what to do with them once they’ve been identified.