Python split string on whitespace and punctuation

I'm trying to split a string up into words and punctuation, adding the punctuation to the list produced by the split.

For instance:

>>> c = "help, me"
>>> print c.split()
['help,', 'me']

What I really want the list to look like is:

['help', ',', 'me']

So, I want the string split at whitespace with the punctuation split from the words.

I've tried to parse the string first and then run the split:

>>> for character in c:
...     if character in ".,;!?":
...             outputCharacter = " %s" % character
...     else:
...             outputCharacter = character
...     separatedPunctuation += outputCharacter
>>> print separatedPunctuation
help , me
>>> print separatedPunctuation.split()
['help', ',', 'me']

This produces the result I want, but is painfully slow on large files.

Is there a way to do this more efficiently?

Fionnuala

89.7k7 gold badges108 silver badges145 bronze badges

asked Dec 14, 2008 at 23:30

1

This is more or less the way to do it:

>>> import re
>>> re.findall(r"[\w']+|[.,!?;]", "Hello, I'm a string!")
['Hello', ',', "I'm", 'a', 'string', '!']

The trick is, not to think about where to split the string, but what to include in the tokens.

Caveats:

• The underscore (_) is considered an inner-word character. Replace \w, if you don't want that.
• This will not work with (single) quotes in the string.
• Put any additional punctuation marks you want to use in the right half of the regular expression.
• Anything not explicitely mentioned in the re is silently dropped.

answered Dec 15, 2008 at 1:53

4

Here is a Unicode-aware version:

re.findall(r"\w+|[^\w\s]", text, re.UNICODE)

The first alternative catches sequences of word characters (as defined by unicode, so "résumé" won't turn into ['r', 'sum']); the second catches individual non-word characters, ignoring whitespace.

Note that, unlike the top answer, this treats the single quote as separate punctuation (e.g. "I'm" -> ['I', "'", 'm']). This appears to be standard in NLP, so I consider it a feature.

answered Jan 19, 2012 at 17:58

LaCLaC

12.5k5 gold badges37 silver badges38 bronze badges

1

If you are going to work in English (or some other common languages), you can use NLTK (there are many other tools to do this such as FreeLing).

import nltk
nltk.download('punkt')
sentence = "help, me"
nltk.word_tokenize(sentence)

sh37211

1,2611 gold badge15 silver badges34 bronze badges

answered Nov 8, 2018 at 16:16

1

Here's my entry.

I have my doubts as to how well this will hold up in the sense of efficiency, or if it catches all cases (note the "!!!" grouped together; this may or may not be a good thing).

>>> import re
>>> import string
>>> s = "Helo, my name is Joe! and i live!!! in a button; factory:"
>>> l = [item for item in map(string.strip, re.split("(\W+)", s)) if len(item) > 0]
>>> l
['Helo', ',', 'my', 'name', 'is', 'Joe', '!', 'and', 'i', 'live', '!!!', 'in', 'a', 'button', ';', 'factory', ':']
>>>

One obvious optimization would be to compile the regex before hand (using re.compile) if you're going to be doing this on a line-by-line basis.

answered Dec 15, 2008 at 1:30

1

Here's a minor update to your implementation. If your trying to doing anything more detailed I suggest looking into the NLTK that le dorfier suggested.

This might only be a little faster since ''.join() is used in place of +=, which is known to be faster.

import string

d = "Hello, I'm a string!"

result = []
word = ''

for char in d:
    if char not in string.whitespace:
        if char not in string.ascii_letters + "'":
            if word:
                    result.append(word)
            result.append(char)
            word = ''
        else:
            word = ''.join([word,char])

    else:
        if word:
            result.append(word)
            word = ''
print result
['Hello', ',', "I'm", 'a', 'string', '!']

answered Dec 15, 2008 at 1:05

monkutmonkut

40.3k23 gold badges118 silver badges148 bronze badges

3

This worked for me

import re

i = 'Sandra went to the hallway.!!'
l = re.split('(\W+?)', i)
print(l)

empty = ['', ' ']
l = [el for el in l if el not in empty]
print(l)

Output:
['Sandra', ' ', 'went', ' ', 'to', ' ', 'the', ' ', 'hallway', '.', '', '!', '', '!', '']
['Sandra', 'went', 'to', 'the', 'hallway', '.', '!', '!']

answered Apr 21, 2020 at 8:41

MalgoMalgo

1,5331 gold badge14 silver badges28 bronze badges

I think you can find all the help you can imagine in the NLTK, especially since you are using python. There's a good comprehensive discussion of this issue in the tutorial.

answered Dec 15, 2008 at 0:34

dkretzdkretz

37.1k13 gold badges79 silver badges136 bronze badges

I came up with a way to tokenize all words and \W+ patterns using \b which doesn't need hardcoding:

>>> import re
>>> sentence = 'Hello, world!'
>>> tokens = [t.strip() for t in re.findall(r'\b.*?\S.*?(?:\b|$)', sentence)]
['Hello', ',', 'world', '!']

Here .*?\S.*? is a pattern matching anything that is not a space and $ is added to match last token in a string if it's a punctuation symbol.

Note the following though -- this will group punctuation that consists of more than one symbol:

>>> print [t.strip() for t in re.findall(r'\b.*?\S.*?(?:\b|$)', '"Oh no", she said')]
['Oh', 'no', '",', 'she', 'said']

Of course, you can find and split such groups with:

>>> for token in [t.strip() for t in re.findall(r'\b.*?\S.*?(?:\b|$)', '"You can", she said')]:
...     print re.findall(r'(?:\w+|\W)', token)

['You']
['can']
['"', ',']
['she']
['said']

answered Apr 15, 2014 at 19:11

FrauHahnhenFrauHahnhen

1332 silver badges11 bronze badges

Try this:

string_big = "One of Python's coolest features is the string format operator  This operator is unique to strings"
my_list =[]
x = len(string_big)
poistion_ofspace = 0
while poistion_ofspace < x:
    for i in range(poistion_ofspace,x):
        if string_big[i] == ' ':
            break
        else:
            continue
    print string_big[poistion_ofspace:(i+1)]
    my_list.append(string_big[poistion_ofspace:(i+1)])
    poistion_ofspace = i+1

print my_list

Aurasphere

3,68112 gold badges42 silver badges69 bronze badges

answered Apr 18, 2017 at 9:03

Have you tried using a regex?

http://docs.python.org/library/re.html#re-syntax

By the way. Why do you need the "," at the second one? You will know that after each text is written i.e.

[0]

","

[1]

","

So if you want to add the "," you can just do it after each iteration when you use the array..

answered Dec 14, 2008 at 23:34

Filip EkbergFilip Ekberg

35.8k19 gold badges123 silver badges181 bronze badges

In case you are not allowed to import anything,use this!

word = "Hello,there"
word = word.replace("," , " ," )
word = word.replace("." , " .")
return word.split()

Kosuke Sakai

2,1912 gold badges4 silver badges11 bronze badges

answered Nov 27, 2019 at 9:14

How do you split a string by punctuation and spaces in Python?

How do you split a string with spaces and commas?

How do you split a string in whitespace in Python?

Can I split a string by two delimiters Python?