Imbedded special characters

Imbedded special characters

fanac 13 Jul 2011 16:43

While downloading files pages from our Fancyclopedia site I ran into a problem with imbedded special characters that were keyed or copied into some (about 10% of them so far) of them while they were being initially entered.

page = s.pages.get_one({'site': 'fancyclopedia', 'page': file_name})     
print page
 
{'commented_at': None, 'rating': 0, 'updated_by': 'mlo1', 'title': 'Anticipation', 'created_at': '2011-06-29T00:24:41+00:00', 'title_shown': 'Anticipation', 'updated_at': '2011-06-29T00:24:54+00:00', 'created_by': 'mlo1', 'children': 0, 'content': u'The 2009 [[[Worldcon]]] held in the Palais des congr\xe8s de Montr\xe9al.  [[[GoHs]]]: [[[Neil Gaiman]]] (pro)
.
.
.
,
n-worldcons">Canadian Worldcons</a>.</p>\n', 'commented_by': None, 'comments': 0, 'parent_title': None, 'fullname': 'anticipation', 'parent_fullname': None, 'tags': ['worldcon', 'convention', 'canada'], 'revisions': 1}
 
content = page['content']
print content
 
The 2009 [[[Worldcon]]] held in the Palais des congrès de Montréal.  [[[GoHs]]]: [[[Neil Gaiman]]] (pro), [[[Elisabeth Vonarburg]]] (pro), [[[David Hartwell]]] (editor), [[[Tom Doherty]]] (publisher) and [[[FGoH]]]: [[[Taral Wayne]]]. [[[Chairmen]]]: [[[René Walling]]] and [[[Robbie Bourget]]].
 
[[[Ralph Bakshi]]] was originally announced as Artist [[[GoH]]] but withdrew for health reasons.
 
See also [[[Canadian Worldcons]]].
 
temp.write(content)
 
Traceback (most recent call last):
  File "D:\My Stuff\Wikidot\chkfiles.py", line 35, in <module>
    temp.write(content)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 52: ordinal not in range(128)

There are both the '\x' with two hex characters and a '\u' form with four hex characters and both of them caused failures, the '\u' form failed when trying to print.

Traceback (most recent call last):
  File "D:\My Stuff\Wikidot\chkfiles.py", line 31, in <module>
    print content
  File "C:\Python27\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u2022' in position 36: character maps to <undefined>

Is there a way to get around this without having to manually correct the existing files?

Thanks for your help…

---

Jack Weaver Fanac Fan History Project

Last edited on 14 Jul 2011 09:09 by fanac Show more

Re: Imbedded special characters

leiger 14 Jul 2011 02:36

I don't know for sure, but it looks like you're trying to parse a unicode character using an ascii codec. Unicode includes all 128 ASCII characters + more.

Perhaps try explicitly specifying to your XML-RPC library that you want to parse using unicode?

Other than that, I'm not sure what to suggest.

---

Also, just a quick tip about terminology… the pages.get_one method downloads pages, not files (you need the files.get_one method for that). When I initially read "While downloading files from our Fancyclopedia site…" I thought you were talking about something completely different :)

Think of a Wikidot page as an Object in object-oriented programming - it's not actually a HTML page.

---

~ Shane (Wikidot Community Admin - Volunteer)
Wikidot: Wikidot Editor, Official Docs
Other: YouTube (gaming, primarily Minecraft)

Re: Imbedded special characters

soronlin 14 Jul 2011 07:28

Yes, it is to do with Unicode encodings when writing the file.

To get around this you will need to write the file as Unicode, probably UTF-8. It's pretty simple to do, once you get your head around the codecs library. Here's a program I wrote earlier that handles (UTF-16) Unicode on input and output:

import codecs
 
for i in range(2,25):
    with codecs.open("K1.xml", encoding="utf-16le") as fin:
        with codecs.open("K"+str(i)+".xml", "w+", encoding="utf-16le") as fout:
            for line in fin:
                fout.write(line.replace("K1", "K"+str(i)))

Re: Imbedded special characters

fanac 14 Jul 2011 09:37

Sorry, I'm new to WikiDot and so I'm not up on it's terminology. I meant pages.

Does WikiDot support the direct use of characters that are outside the '10'X - '7F'X range? Personally I would doubt it, I know that their use is discouraged in HTML. I think the use of @>&----;<@ for them in WikiDot pages would be pretty much required.

What I need to do is fix the problem by replacing these with the correct form.

---

Jack Weaver Fanac Fan History Project

Re: Imbedded special characters

soronlin 14 Jul 2011 10:58

I know that their use is discouraged in HTML.

Do you? I don't know why. The HTML 4.01 Specification says this.

The ASCII character set is not sufficient for a global information system such as the Web, so HTML uses the much more complete character set called the Universal Character Set (UCS), defined in [ISO10646]. This standard defines a repertoire of thousands of characters used by communities all over the world.

The character set defined in [ISO10646] is character-by-character equivalent to Unicode ([UNICODE]).
…
Authoring tools (e.g., text editors) may encode HTML documents in the character encoding of their choice, and the choice largely depends on the conventions used by the system software. These tools may employ any convenient encoding that covers most of the characters contained in the document, provided the encoding is correctly labeled. Occasional characters that fall outside this encoding may still be represented by character references. These always refer to the document character set, not the character encoding.

Servers and proxies may change a character encoding (called transcoding) on the fly to meet the requests of user agents (see section 14.2 of [RFC2616], the "Accept-Charset" HTTP request header). Servers and proxies do not have to serve a document in a character encoding that covers the entire document character set.

Commonly used character encodings on the Web include ISO-8859-1 (also referred to as "Latin-1"; usable for most Western European languages), ISO-8859-5 (which supports Cyrillic), SHIFT_JIS (a Japanese encoding), EUC-JP (another Japanese encoding), and UTF-8 (an encoding of ISO 10646 using a different number of bytes for different characters). Names for character encodings are case-insensitive, so that for example "SHIFT_JIS", "Shift_JIS", and "shift_jis" are equivalent.

This specification does not mandate which character encodings a user agent must support.

Conforming user agents must correctly map to ISO 10646 all characters in any character encodings that they recognize (or they must behave as if they did).

By the way, printable characters in ASCII are 0x20-0x7E, not 0x10-0x7F.

Last edited on 14 Jul 2011 11:01 by soronlin Show more

Re: Imbedded special characters

fanac 14 Jul 2011 12:08

I don't remember where I saw it years ago but I do remember an article that strongly recommended that the ISO Entities be used instead of single characters that are above 0x7F to avoid ambiguity and, at least to me, that makes a great deal of sense.

I humbly stand corrected, 0x20-0x7F is actual range, but that doesn't provide any help to me in my present situation because somewhere something failed to "correctly map to ISO 10646 all characters in any character encodings that they recognize", probably because it didn't recognize the damn darn things.

---

Jack Weaver Fanac Fan History Project

Re: Imbedded special characters

soronlin 14 Jul 2011 12:43

0x7E ;-)

It was your Python code that failed to "correctly map to ISO 10646 all characters in any character encodings that they recognize". The XML-RPC library correctly received those characters from Wikidot.com and saved them correctly in a Unicode string. You then tried to save them in an ASCII file and Python correctly reported that doing so was impossible. Use the codecs Python library as I demonstrated above and encode the file as UTF-8. It will add one and a half a lines to your code and the resulting file will be readable as ASCII where it does not have any esoteric characters in.

Or you could use (one of) the "latin_1", "cp1252" or "cp850" codecs instead of a UTF one. That will encode most Western European languages into 8-bit "extended ASCII".

Re: Imbedded special characters

fanac 14 Jul 2011 15:22

I can't ague with you re: when the problem lies. It also lies in the fact that while I'm not new to programming I am a newbie to Python.

Unfortunately your code snippet assumes the existence of a readable file for "fin" and my problem is that, because of the errors mentioned earlier, I cannot write the raw data that needs to be corrected to disk in the first place…

page = s.pages.get_one({'site': 'fancyclopedia', 'page': 'ace-double'})
content = page['content']

Is there a way to massage "content" without first writing it to disk?

I'm running Windows XP and using Python 2.7.2 if that helps.

Thanks for trying…

---

Jack Weaver Fanac Fan History Project

Re: Imbedded special characters

soronlin 14 Jul 2011 15:45

The example I posted earlier was nothing to do with Wikidot. It was merely an example of the use of codecs both for input and output.

Here is how to do the job you seem to want to do — read a page and save it to a file. I assume you are on Windows and that the standard Windows codepage 1252 is appropriate. If you get the wrong characters when viewing the file in Notepad or whatever, try some of the other codecs I listed above.

Your listed code does not show how you open the file temp, so I will make the assumption that it is just a file called "tempfile.txt" and you wish to silently replace any existing content.

You will also need to add in the stuff that opens the XML-RPC connection s, but you know that.

import codecs
 
with codecs.open("tempfile.txt", "w+", encoding="cp1252") as temp:
    page = s.pages.get_one({'site': 'fancyclopedia', 'page': 'ace-double'})
    content = page['content']
    temp.write(content)

Re: Imbedded special characters

fanac 14 Jul 2011 17:00

This is the test script (testing.py)

from xmlrpclib import ServerProxy
import codecs
 
{snip}
 
s = ServerProxy('https://fancyclopedia:' + key[0].rstrip('\n') +'@www.wikidot.com/xml-rpc-api.php')
target = open('testing.txt', 'w')
page = s.pages.get_one({'site': 'fancyclopedia', 'page': 'ace-double'})
content = page['content']
print content
print '\n'
target.write(content)
target.close()

and this is it's output

The Ace Doubles were a series of books published by [[[Ace Books]]], mostly under the editorship of [[[Donald A. Wollheim]]] from the early 50s to the late 70s. They consisted of two [[[novellas]]] or short novels (typically 75-125 printed pages bound tête-bêche, a format in which the book has two stories bound upside-down with respect to each other, with two front covers and the end of the two stories meet in the middle of the book.

The stories were often taken from the [[[British SF magazines]]] and were frequently heavily edited.

Traceback (most recent call last):
  File "D:\My Stuff\Wikidot\testing.py", line 14, in <module>
    target.write(content)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 253: ordinal not in range(128)

The problem is caused by "tête-bêche" in the 1st line of the output and it was probably entered manually since it's not part of Fancy 2.

As you can see I cannot write to a file…

---

Jack Weaver Fanac Fan History Project

Re: Imbedded special characters

soronlin 14 Jul 2011 17:28

You need to use the codecs library to open the file.

Since I couldn't access your site with my key, I copied it over to my site for testing. The only thing you need to change is the "target=…" line.

from xmlrpclib import ServerProxy
import codecs

key=["xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"]

s = ServerProxy('https://soronlin-test:' + key[0].rstrip('\n')
+'@www.wikidot.com/xml-rpc-api.php')
target = codecs.open('testing.txt', 'w',"cp1252")
page = s.pages.get_one({'site': 'soronlin-test', 'page': 'ace-double'})
content = page['content']
print content
print '\n'
target.write(content)
target.close()

Resulting in the output:

richard@jupiter ~/development $ python test.py
The Ace Doubles were a series of books published by [[[Ace Books]]], mostly under the editorship of [[[Donald A. Wollheim]]] from the early 50s to the late 70s. They consisted of two [[[novellas]]] or short novels (typically 75-125 printed pages bound tête-bêche, a format in which the book has two stories bound upside-down with respect to each other, with two front covers and the end of the two stories meet in the middle of the book.

The stories were often taken from the [[[British SF magazines]]] and were frequently heavily edited.

And if I have chosen the right codec (I'm on Linux now, so CP1252 is a guess as to what Windows will find acceptable, but I can configure my command window to use it) typing the file shows the right characters. ( cat is the Linux command that does the same as the MS command type )

richard@jupiter ~/development $ cat testing.txt
The Ace Doubles were a series of books published by [[[Ace Books]]], mostly under the editorship of [[[Donald A. Wollheim]]] from the early 50s to the late 70s. They consisted of two [[[novellas]]] or short novels (typically 75-125 printed pages bound tête-bêche, a format in which the book has two stories bound upside-down with respect to each other, with two front covers and the end of the two stories meet in the middle of the book.

The stories were often taken from the [[[British SF magazines]]] and were frequently heavily edited.

Last edited on 14 Jul 2011 17:34 by soronlin Show more

Re: Imbedded special characters

fanac 14 Jul 2011 20:51

Thank you. This was a great help, I just finished downloading and processing most of Fancyclopedia's pages (1750+) and only had 81 errors similar to this one.

Traceback (most recent call last):
  File "D:\My Stuff\Wikidot\getfiles.py", line 65, in <module>
    target.write('Content: ' + content + '\n')
  File "C:\Python27\lib\codecs.py", line 691, in write
    return self.writer.write(data)
  File "C:\Python27\lib\codecs.py", line 351, in write
    data, consumed = self.encode(object, self.errors)
  File "C:\Python27\lib\encodings\cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u03bd' in position 20: character maps to <undefined>

I knew that there would be some that had to hand corrected but this is manageable.

Again, thank you.

---

Jack Weaver Fanac Fan History Project

Last edited on 14 Jul 2011 21:00 by fanac Show more

Re: Imbedded special characters

soronlin 15 Jul 2011 00:16

Probably because those characters cannot be represented in CP1252. If you encoded to UTF-8 you would not have even those errors. But fewer programs would read it, it all depends what you need those files for.

Re: Imbedded special characters

leiger 15 Jul 2011 04:11

Thanks rurwin for stepping in, you're the expert on this type of thing after all ;-) Have seen you solve similar problems before, IIRC. I was out of my league here and just posting educated guesses at a solution (which apparently came close).

@jackw: No need to apologise for the pages/files mistake. I only explained it to help clear the definition up for you, as I thought you might be interested :)
If you have any other questions, I'll be happy to help answer them as best I can.

---

~ Shane (Wikidot Community Admin - Volunteer)
Wikidot: Wikidot Editor, Official Docs
Other: YouTube (gaming, primarily Minecraft)

Re: Imbedded special characters

fanac 15 Jul 2011 22:02

I tried it both ways, CP1252and UTF-8, an it seemed to work much the same way in either case. The final error rate 0.7% (a dozen files) and it was pretty obvious where the errors were when I checked a few, the little black diamond with the ? inside…

I did have a very weird error with one page. The page's name is 'con' and when I tried to open 'con.src' this is what happened.

con
Traceback (most recent call last):
  File "D:\My Stuff\Wikidot\getpags.py", line 45, in <module>
    target = codecs.open('Files\\' + page_name + '.src', 'w', "UTF-8")
  File "C:\Python27\lib\codecs.py", line 881, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: 'Files\\con.src'

Thanks again for your help.

---

Jack Weaver Fanac Fan History Project

Re: Imbedded special characters

soronlin 17 Jul 2011 08:55

Well done on finding one of the most obscure features on Windows.

CON is a reserved filename in MS-DOS etc. You will also find that you cannot create a file called PRN or LPT1 either.

Re: Imbedded special characters

fanac 17 Jul 2011 20:28

Leave it to me to step into a obscure pothole, oh well, live and learn… :-}

I understand about PRN & LPT1 but CON? Besides, I wasn't trying to create a file named CON rather I was trying to create a file called CON.SRC… Picky, picky…

I'll just have to fudge it…

---

Jack Weaver Fanac Fan History Project

Re: Imbedded special characters

Ed Johnson 17 Jul 2011 21:02

CON is short for CONsole in DOS-speak. I used it occasionally many moons ago to create quick and dirty batch files. Open a command prompt and try this:

C:\>copy con test.bat
@echo off
echo Hello World!
^Z
C:\>type test.bat
C:\>test.bat
C:\>del test.bat

Anything you type is captured, including CR/LF. Terminate entry with CTRL-Z (^Z) and the file gets written.

Ed

---

Community Admin