Saving Emoji to Backendless

florin-adrian · February 28, 2016, 7:58am

I understood from another topic that because your database is UTF8 it cannot handle emojis. As suggested there, I converted the string to UTF8 and saved it successfully. However, when I try to convert it back to UTF16 to show emojis, it does not work, the emojis are shown as below.
Emoji: ðŸ˜ðŸ˜ðŸ’œðŸ˜‚ðŸ†ŽðŸ˜„ðŸ’œðŸ†ŽðŸ˜‚ðŸ˜…ðŸ’œðŸ†ŽðŸ˜‚ðŸ†ŽðŸ˜¬ðŸ’œðŸ’–ðŸ˜¬ðŸ†ŽðŸ˜
Can you please let me know what I’m doing wrong?
These are the conversion functions.
class func stringToUTF8String (string: String) -> String? {

let encodedData = string.dataUsingEncoding(NSUTF8StringEncoding)!
let attributedOptions = [NSDocumentTypeDocumentAttribute: NSHTMLTextDocumentType]
do{
let attributedString = try NSAttributedString(data: encodedData, options: attributedOptions, documentAttributes: nil)
return attributedString.string
}catch _ {
}

return nil
}

class func stringToUTF16String (string: String) -> String? {

let encodedData = string.dataUsingEncoding(NSUTF16StringEncoding)!
let attributedOptions = [NSDocumentTypeDocumentAttribute: NSHTMLTextDocumentType]
do{
let attributedString = try NSAttributedString(data: encodedData, options: attributedOptions, documentAttributes: nil)
return attributedString.string
}catch _ {
}

return nil
}

mark-piller · February 28, 2016, 6:27pm

This question is much better positioned for something like stackoverflow.com. Try posting it there.

florin-adrian · March 2, 2016, 2:12am

Thanks Mark.

I did ask this on stackoverflow. I tried the advice I got there, I tried a lot of other things as well and I’m still unable to save / display emojis.
I am sure that my lack of knowledge has some role in figuring this out quickly but still, it seems (at least to me) that this is an issue that someone from Backendless should address as it has a serious impact on some type of applications. At least some support in terms of how the conversion should be made.

For the record, if someone from Backendless will try to give me an advice (please do), after trying various ways of converting the string to UTF8, I either end up having some weird strings saved in the DB, such as %F0%9F%99%84%F0%9F%98%80%F0%9F%98%92% or ðŸ¤”ðŸ¤”ðŸ‘…ðŸ™ðŸ˜‚ðŸ˜ðŸ˜ŽðŸ˜‰ðŸ˜ or, in cases where I’m unsuccessful in my attempt to save the object, I get the following error:

FAULT = ‘Server.Processing’ [java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect string value: ‘\xF0\x9F\x98\xB3\xF0\x9F…’ for column ‘text.CD7DC959-5457-443F-FF2C-E1C1F3492300’ at row 1] <java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect string value: ‘\xF0\x9F\x98\xB3\xF0\x9F…’ for column ‘text.CD7DC959-5457-443F-FF2C-E1C1F3492300’ at row 1>

I also found out that emojis should also be supported by UTF8, therefore the string should not be converted back to UTF16 when displaying the text obtained from your DB.

Thanks.

mark-piller · March 2, 2016, 2:29am

Hi Florin,

I do recognize it is a common problem. You’re right, it appears they can be encoded in UTF-8. I will bring it up in our standup meeting tomorrow so we can decide on the resolution and provide a clear path to support emojis.

Regards,
Mark

florin-adrian · March 2, 2016, 2:50am

Thanks

florin-adrian · March 3, 2016, 12:53am

I tried this again, based on the feedback received here: utf 8 - Swift Conversion UTF16 to UTF8 and back - Stack Overflow

I was able to reproduce the results that “hola” got in the stackoverflow topic. I attached a screenshot as well. As you can see I can convert a string to UTF8 and the emojis will still work. However, when I try to save a converted string, with either of the two approaches, I get an error back from Backendless.

FAULT = ‘Server.Processing’ [java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect string value: ‘\xF0\x9F\x98\xAD i…’ for column ‘text.CD7DC959-5457-443F-FF2C-E1C1F3492300’ at row 1] <java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Incorrect string value: ‘\xF0\x9F\x98\xAD i…’ for column ‘text.CD7DC959-5457-443F-FF2C-E1C1F3492300’ at row 1>
reloading data

At this point I don’t see what else I could be doing to get this to work. Correct me if I’m wrong but this seems to be a problem on your end.

mark-piller · March 3, 2016, 4:28am

Does the error occur when you save str1 or str2?

I do not argue that the problem is on our end, indeed, our storage does not handle emoji characters well. We just wanted to see if a temporary solution would work for you.

Regards,
Mark

florin-adrian · March 3, 2016, 4:32am

The same type of error occurs for both str1 and str2.

florin-adrian · March 15, 2016, 2:06am

Is anyone, by any chance, working on a solution to this? At least a temporary one?

Thanks

mark-piller · March 16, 2016, 10:38am

This will be supported eventually in the platform. It is on our roadmap.

florin-adrian · April 7, 2016, 2:39am

Sorry to have to come back again to this. But do you have any idea if emojis will be supported in the next month or two?

mark-piller · April 9, 2016, 6:24pm

It is in QA now, so it is very likely that it will work in the next month of two.

paetur-magnussen · May 22, 2016, 11:17am

Any update on this?

kate-maksimenko · May 25, 2016, 8:19am

Hi!

We added new data type (EXTENDED_STRING) to support emoji in data service: example1 and example2
Regards,
Kate.

paetur-magnussen · May 26, 2016, 8:17am

Thanks!

What is an extended string? Is it unicode?

sergey-kukurudzyak · May 26, 2016, 9:38am

Yes it is utf-8, but extended to store more data

florin-adrian · May 26, 2016, 5:31pm

Thanks for the update.
I just tried this and I don’t understand if I’m doing something wrong or not because it doesn’t work. The text with emojis uploads fine to the database, as I can see it right on the website. However, when I try to download the text in the app, I get a nil value.

And one more note: when I upload the text without any emojis, it works fine. When I add emojis to the text, I get nil back (I’m using Swift, btw).

This is how the class is declared

class Post: NSObject {

var ownerId: String?

var objectId: String?

var created: NSDate?

var updated: NSDate?




var replies: Int = 0

var postText: String?

}

And this is how I try to retrieve the data:

backendless.data.of(Post.ofClass()).find(dataQuery,

        response: { (collection) -> Void in

            

            for post in collection.data as! [Post]{

                print("TEXT: \(post.postText)") //This is where I get nil if the string has emojis

            }

}) { (error) -> Void in
self.refreshControl?.endRefreshing()
}

istvan-sipos · May 27, 2016, 4:31pm

Hello,

I also needed to upload emojis to Backendless a few days ago, and I implemented it by percent encoding the string before uploading it, and decoded it after I downloaded the object from the server.

So I had this getter and setter on my object:

(void)setDetails:(NSString*)details {

self.backendlessObservation.details = [details stringByAddingPercentEncodingWithAllowedCharacters:[NSCharacterSet alphanumericCharacterSet]];

}

(NSString*)details {

return [self.backendlessObservation.details stringByRemovingPercentEncoding];

}

The first (setter) encodes the string before saving the object to Backendless and the second (getter) decodes the string after the object is fetched from the server.

However, I didn’t observe that there is an Extended String, so I will remove my workaround and will give it a try to see how it works.

Regards,

Istvan

vyacheslav-vdovichenko · June 10, 2016, 3:33pm

Please update the libs from github (CoomLibiOs & backendless), and try again

florin-adrian · June 11, 2016, 6:58am

Same thing happens